10. Personalized Headshot Generation

GitHub에서 수정

A user uploads a dozen phone photos and later receives fifty studio-style headshots of themselves in a suit, under soft light, against a blurred office. A pretrained text-to-image model can draw “a professional headshot of a man” all day, but it has never seen this face, and no prompt describes a face precisely enough to recover it. The system has to teach the model a new identity from a tiny, noisy dataset, once per customer.

The core tension: train hard enough to reproduce the exact jawline and eye spacing, and the model forgets how to draw anything else. Every “person” turns into the customer, poses freeze into the selfies, and text control over the scene fades. Hold back and you get a well-lit stranger who resembles the customer the way a sibling does, which customers spot instantly. The design is about where that line sits, which personalization method reaches it inside the time and cost budget, and how every output gets checked before a human sees it.

The notes follow the book’s seven-step framework, then add follow-ups, interview questions, a 45-minute walkthrough and review questions.

1. Clarifying requirements

Outputs go on LinkedIn, so the scene vocabulary is small. What I’d settle first:

• Photo count. One reference forces a tuning-free encoder. Ten to twenty varied photos make per-user finetuning practical.
• Who writes prompts. If users don’t, the team owns a template library, which removes input-side prompt safety problems and can be tuned offline.
• Attribute control. Out of scope means no editing models or attribute encoders.
• Count and resolution. Fifty images at 1024x1024 fixes an SDXL-class base and dominates the sampling bill.
• Bad uploads. Dark, blurry, occluded or multi-person photos must be caught before training.
• Latency. “Email within the hour” makes this a batch job and allows a 15 minute finetune. “Results in 10 seconds” would rule per-user training out.
• Retention. If the personalized model is deleted after delivery, per-user storage is close to free.
• Privacy. Face photos are biometric data under GDPR and laws like Illinois BIPA, so consent, retention and deletion are hard requirements.

2. Framing the problem as an ML task

Input: photos \( \{x_1,\dots,x_n\} \) of one person, \( n\approx 10\text{–}20 \). Output: 50 images that are photorealistic, match a professional scene description, depict the same person, and differ in pose, lighting and setting. It is conditional image generation with one condition, identity, that text cannot express.

A pretrained latent diffusion model is the natural generator. StyleGAN-type GANs draw sharp aligned portraits but follow free-form text poorly, and autoregressive image-token models are slower at 1024px. The real question is how to inject identity into the diffusion model.

Tuning-free versus tuning-based

Tuning-free methods train an image encoder once, offline, and inject its output (often a face-recognition embedding plus CLIP image features) through extra cross-attention or adapter layers. Public examples: IP-Adapter-FaceID, InstantID, PhotoMaker, Meta’s Imagine Yourself. One shared model answers in seconds, but one embedding drops details seen only at other angles, and the encoder needs large same-person paired data.

Tuning-based methods modify the model per user. They differ in which parameters move:

• Textual inversion learns one new token embedding \( v_* \). Kilobytes, base untouched, but a single vector in word-embedding space can’t carry fine facial geometry.
• DreamBooth finetunes all U-Net weights (optionally the text encoder) and binds the subject to an identifier token. Best fidelity from few images; a full model copy per user.
• LoRA freezes the base and learns low-rank updates on chosen linear layers. Megabyte adapters, far less training memory, face quality close to full DreamBooth. The DreamBooth objective and LoRA parameterization combine freely, and “DreamBooth-LoRA” is common in production.

I’d pick full DreamBooth. Identity is the product, an hour fits a 15 minute finetune, and discarding weights erases the storage penalty. If the product later offered “come back for new styles”, I’d switch to DreamBooth-LoRA and keep a small adapter per user.

The identifier token

Prompts need a word for the subject. A common word (“john”) already has meaning that leaks into images. A random string (“x7qzp9”) gets split by the BPE tokenizer into pieces that each carry associations. A token that is a single vocabulary unit but was rare in pretraining (“sks” in CLIP’s tokenizer is the community default) has a weak prior and behaves like an empty slot. Pairing it with a class noun (“a photo of sks man”) lets the model reuse what it knows about the class and learn only the difference.

3. Data preparation

There is no corpus to curate. The data work is gatekeeping uploads, building a class prior set, and keeping an evaluation panel.

Upload validation

A wasted finetune costs GPU time, so validation runs on cheap hardware first:

• Face detection (RetinaFace or MTCNN): exactly one prominent face. Group photos are the classic silent poison.
• Quality heuristics: variance of the Laplacian on the face crop for blur, luminance histogram for exposure, a minimum face width of about 256 px, an occlusion classifier for sunglasses, masks and hands.
• Identity consistency: embed faces with an ArcFace-style recognition model, compute pairwise cosine similarity, keep the dominant cluster. Outliers are another person or an extreme angle.
• Diversity: estimate head yaw and pitch; flag sets of near-identical frontal selfies.
• Policy: age estimate, nudity check, and a live selfie match to confirm the uploader is the subject.

If fewer than 10 to 15 images survive, the user gets a specific request (“4 more, at least one in profile”).

Preprocessing and class images

Crop around the detected face with headroom and shoulders and resize to 1024; blind center crops cut chins. Augment lightly (small rotations, scale and color jitter). Captioning incidental content (“red hoodie”) tells the model the hoodie is not identity. The VAE is frozen, so encode each image to a latent once.

Prior preservation needs a few hundred to a thousand generic images sampled from the frozen base model with “a photo of a man”. Sampling from the base, rather than using real photos, anchors the finetuned model to its own earlier behaviour, which is what we want to keep. These depend only on base version and class noun, so generate them once and cache the latents.

    flowchart LR
    accTitle: Upload validation and data preparation pipeline
    accDescr: User photos pass face detection, quality checks and identity clustering; failing sets trigger a request for more photos; passing sets are cropped, augmented and encoded to latents, then joined with cached class latents generated once by the frozen base model.
    UP["User uploads 10-20 photos"] --> FD["Face detection: one face"]
    FD --> QH["Blur, exposure, size, occlusion"]
    QH --> IC["Identity clustering"]
    IC --> DV{"Enough good, varied photos?"}
    DV -- no --> ASK["Request specific extra photos"]
    DV -- yes --> CR["Face-aware crop to 1024"]
    CR --> AUG["Light augmentation and captions"]
    AUG --> ENC["VAE encode to latents"]
    BASE[("Frozen base model")] --> CP["Generate class images once"]
    CP --> CACHE[("Cached class latents")]
    ENC --> JOB["Training job bundle"]
    CACHE --> JOB
  

Pitfalls: an accessory in every photo becomes part of the identity, and a face detector that is weaker on darker skin or older faces rejects those users more often, so track acceptance by subgroup.

4. Model development

Architecture

The base is latent diffusion with three parts:

1. A frozen VAE maps 1024x1024x3 images to 128x128x4 latents and back. Working in latent space is what makes 1024px affordable.
2. A text encoder (CLIP ViT-L, plus OpenCLIP ViT-bigG in SDXL) turns the prompt into token embeddings. Frozen by default.
3. A U-Net \( \epsilon_\theta(z_t,t,c) \): residual conv blocks at several resolutions with skip connections, transformer blocks holding spatial self-attention and cross-attention to text, and a sinusoidal timestep embedding fed to every block. SDXL’s U-Net has about 2.6B parameters. This is what DreamBooth trains.

The identifier acts through cross-attention:

\( h \) are spatial features, \( c \) the text token embeddings, and \( d \) the head dimension. The key and value of “sks” decide what gets written into the image positions that attend to it, so tuning \( W_K \), \( W_V \) and the following layers is how the token comes to mean this face. It is also why LoRA restricted to attention projections captures most of the gain.

For a frozen weight \( W_0\in\mathbb R^{d_{out}\times d_{in}} \), LoRA learns

\( r \) is the rank, \( \alpha \) a scale, \( A \) starts random and \( B \) starts at zero so training begins from the exact base model. Trainable parameters per layer fall from \( d_{out}d_{in} \) to \( r(d_{in}+d_{out}) \), and the update can be merged into \( W_0 \) for zero inference overhead.

Worked example: adapter versus checkpoint. A 1280x1280 projection has 1,638,400 weights. At \( r=16 \), LoRA trains \( 16\times(1280+1280)=40{,}960 \), or 2.5%. Over all attention projections an SDXL adapter at rank 16 to 32 is tens of MB in fp16, against \( 2.6\text{B}\times 2 \) bytes \( \approx 5.2 \) GB for the full U-Net. For 100,000 returning users that is a few TB of adapters versus about 520 TB of checkpoints.

    flowchart TB
    accTitle: Personalized latent diffusion model internals
    accDescr: The prompt with the identifier is encoded by a frozen text encoder; the noisy latent and time embedding pass through U-Net down, middle and up blocks whose cross-attention reads the text tokens; DreamBooth or LoRA updates touch these weights; predicted noise drives a sampler step and the VAE decodes the final latent.
    P["Prompt: headshot of sks man"] --> TE["Text encoder, frozen"]
    TE --> TOK["Token embeddings incl. sks"]
    Z["Noisy latent 128x128x4"] --> DOWN["Down blocks: ResNet and transformer"]
    T["Timestep"] --> TEMB["Time embedding"]
    TEMB --> DOWN
    DOWN --> MID["Middle block"]
    MID --> UPB["Up blocks"]
    DOWN -. skip connections .-> UPB
    TOK --> XA["Cross-attention K and V from text"]
    XA --> DOWN
    XA --> UPB
    LORA["DreamBooth weights or LoRA B·A"] -.-> XA
    UPB --> EPS["Predicted noise"]
    EPS --> STEP["Sampler step"]
    STEP --> VAE["VAE decoder after last step"]
  

Knobs: training the text encoder too (tighter binding, faster overfitting, needs a lower LR), LoRA rank and targets (rank 4 to 8 on attention misses detail; 32 to 64 plus convs nears full finetuning), and above all the base model, since one that draws teeth badly still does after DreamBooth.

Training

Each order runs a short finetune with the standard diffusion objective. A clean latent is noised in closed form:

\( \bar\alpha_t \) is the cumulative product of the noise schedule, near 1 at small \( t \) (almost clean) and near 0 at large \( t \) (almost pure noise). The full DreamBooth loss is

The first term uses latents of the user’s photos with \( c_{\text{sks}} \), the encoding of “a photo of sks man”, and pulls the identifier toward the face. The second uses cached class latents \( z^{pr} \) with \( c_{\text{cls}} \), the encoding of “a photo of a man”, and \( \lambda \) (often 1) sets how much the model is penalized for changing how it draws generic men. Because the class images came from the frozen model, the second term is self-distillation toward the original behaviour.

The two failure directions map onto \( \lambda \) and step count. Too little prior or too many steps causes language drift (“a man” returns the customer) and mode collapse (outputs repeat training poses, background prompts stop working). Too much prior or too few steps gives soft faces that look related but not identical.

    flowchart LR
    accTitle: DreamBooth training step with prior preservation
    accDescr: Each step pairs a user latent captioned with the identifier and a cached class latent captioned with the class noun; both are noised and passed through the shared U-Net; the instance loss and weighted prior loss are summed for an AdamW update, and after a fixed step budget a quick probe checks identity and class drift.
    UL[("User latents")] --> IB["Caption: a photo of sks man"]
    CL[("Class latents")] --> CB["Caption: a photo of a man"]
    IB --> N1["Noise at timestep t"]
    CB --> N2["Noise at timestep t'"]
    N1 --> UN["Shared U-Net"]
    N2 --> UN
    UN --> LI["Instance MSE"]
    UN --> LP["Prior MSE times lambda"]
    LI --> SUM["Total loss"]
    LP --> SUM
    SUM --> OPT["AdamW step, bf16"]
    OPT --> CHK{"Step budget reached?"}
    CHK -- no --> IB
    CHK -- yes --> VAL["Probe: identity score and class drift"]
  

Settings and efficiency

• Steps: overfitting starts within a few hundred steps at 15 images. A common heuristic is around 80 to 100 steps per image at batch 1 to 2, then a fixed budget tuned on held-out users, plus a cheap end-of-run probe (4 samples, identity score, one class prompt).
• Learning rate: about 1e-6 to 5e-6 for full U-Net, around 1e-4 for LoRA, lower still for the text encoder.
• Precompute VAE latents and text embeddings; prompts are fixed per run, so steps only run the U-Net.
• Parallelism: a single job is small, so throughput comes from one job per GPU, not data parallelism. Keep base weights warm on the GPU and restore them from pinned host memory after each job instead of re-reading 5 GB from storage.

Worked example: memory for full DreamBooth. Mixed-precision AdamW holds fp32 master weights (4 B), gradients (2 to 4 B) and two moments (8 B), roughly 16 bytes per parameter: \( 2.6\text{B}\times16\approx42 \) GB before activations. Attention activations on 128x128 latents add tens of GB, so gradient checkpointing is needed on an 80 GB card. 8-bit Adam drops moments from 8 to 2 bytes, saving about 15 GB. LoRA keeps a frozen bf16 U-Net (5.2 GB) and a tiny optimizer state, so a 24 GB GPU suffices.

Symptoms and fixes: copied selfies (fewer steps, higher \( \lambda \)); a face like a relative (more steps, train the text encoder); a hoodie in every output (caption it); burnt colors (lower LR and guidance).

Sampling

Users don’t write prompts. A versioned library of about fifty templates fixes scene, lighting, attire, expression and head angle, each containing the identifier (“professional headshot of sks man, charcoal suit, soft window light, slight smile, blurred office”). Templates are the main lever for diversity and the first place to fix systematic defects: if outdoor templates keep warping ears, rewrite or drop them.

Sampling starts from Gaussian noise in latent space, runs a solver such as DPM-Solver++ for \( N \) steps, and decodes with the VAE. Classifier-free guidance mixes conditional and unconditional predictions each step:

\( \varnothing \) is the empty or negative prompt and \( w \) the guidance scale. Higher \( w \) enforces the prompt and identity more strongly but saturates color and reduces variety. Personalized models usually want a lower \( w \) than the base, around 4 to 7, because finetuning already pushes hard toward the subject.

Each candidate then passes a quality gate: one face at reasonable pose, identity cosine to the reference embeddings above a calibrated threshold, maximum similarity to any upload below a copy threshold, an artifact classifier for hands, teeth and eyes, and a safety classifier. Failures regenerate with the same template and a new seed, capped at about 3 tries, after which a reserve template replaces it.

Worked example: sampling time per order. On an A100 an SDXL-class U-Net at 1024px takes about 60 to 80 ms per step with CFG batched as two. At 30 steps that is about 2 to 2.5 s per image plus 0.3 s VAE decode. Fifty images take about 2 minutes; with 20% regenerations, about 2.5 minutes. Batching four prompts per forward pass adds 20 to 40% throughput.

5. Evaluation

Personalization adds an axis that generic metrics ignore. I score four properties separately because they trade off: prompt alignment, image quality, identity, and diversity.

Prompt alignment uses CLIPScore against the template:

\( E_I \), \( E_T \) are CLIP’s image and text encoders, and \( w \) rescales (2.5 in the original paper). It checks that “blurred office” happened, nothing about who is in it.

Image quality uses FID between Inception features of generated and real sets:

\( \mu \), \( \Sigma \) are feature means and covariances for generated (\( g \)) and real (\( r \)) images. FID needs thousands of samples, so it is a system-level metric: pool outputs across an evaluation panel and compare to real professional headshots. It cannot judge a single user’s 50 images.

Identity is the cosine similarity \( \frac{f(g)\cdot f(r)}{\|f(g)\|\|f(r)\|} \) between embeddings of generated and real photos, and the encoder \( f \) defines what “similar” means. CLIP embeddings match semantics, so two men in suits look alike. DINOv2 is more sensitive to color and texture but still dominated by lighting and clothing. Face recognition embeddings (ArcFace, trained with an angular margin loss to separate identities) are built for “same person”, so they are the primary metric.

Diversity and copying. A model that pastes training photos scores perfectly on identity. Track mean pairwise LPIPS across the 50 outputs and each output’s maximum similarity to any upload.

For human evaluation, consenting panelists rate “this is me” while other raters judge quality; side-by-side comparisons beat absolute scores. Online, ratings skew toward unhappy users, so pair them with downloads per order, refund and regeneration-request rates, and trial-to-paid conversion.

6. Overall ML system design

Three asynchronous stages joined by a queue: intake and validation, per-user training, then generation and delivery. It is a throughput and cost problem with a one-hour SLA, not a millisecond latency problem.

    flowchart LR
    accTitle: End-to-end headshot service architecture
    accDescr: The web app stores uploads in encrypted storage and calls a validation service; valid orders go to a job queue; a GPU worker with the warm base model and cached class latents finetunes and samples from templates; a quality gate requests regenerations; accepted images are watermarked and stored, the user gets a signed link, and a cleanup job deletes data after retention.
    APP["Web app and payment"] --> RAW[("Encrypted upload store")]
    RAW --> VAL["Validation service"]
    VAL --> Q["Job queue with deadlines"]
    Q --> W["GPU worker"]
    BASE[("Base model and class latents")] --> W
    W --> FT["Finetune per user"]
    FT --> GEN["Sample from templates"]
    TPL[("Template library")] --> GEN
    GEN --> QG{"Identity, copy, artifact, safety gate"}
    QG -- fail --> GEN
    QG -- pass --> WM["Watermark and provenance"]
    WM --> OUT[("Output store")]
    OUT --> MAIL["Email signed link"]
    OUT --> DEL["Retention cleanup"]
    RAW --> DEL
  

    sequenceDiagram
    accTitle: Order lifecycle from upload to delivery
    accDescr: The user uploads and pays, the validator accepts the set, the order is queued, a GPU worker trains and generates with retries, stores watermarked outputs and discards the weights, the user is emailed, and uploads are deleted after the retention window.
    participant U as User
    participant A as Web app
    participant V as Validator
    participant Q as Queue
    participant G as GPU worker
    participant S as Storage
    U->>A: Upload 15 photos and pay
    A->>V: Validate set
    V-->>A: 13 usable, one identity
    A->>Q: Enqueue order
    A-->>U: Accepted, ready within the hour
    Q->>G: Lease job
    G->>S: Read user and class latents
    G->>G: Finetune about 15 min
    G->>G: Sample 50, regenerate failures
    G->>S: Write watermarked outputs
    G->>Q: Ack, discard weights
    A->>U: Email download link
    S->>S: Delete uploads after retention
  

Train and sample on the same worker, since shipping a 5 GB checkpoint costs more than sampling. Autoscale on queue age, not GPU utilization; spot GPUs are fine if jobs checkpoint after training. Make jobs idempotent so a crashed worker’s order can be re-leased without double charging, and run validation on cheap hardware.

Cost levers in order of payoff: cached class and VAE latents (free), batching prompts, a better solver with fewer steps (detail drops below about 20), DreamBooth-LoRA on cheaper GPUs, and fp8 or int8 inference after a quality check. Step-distilled 4 to 8 step bases are tempting but personalize less reliably.

Worked example: fleet and storage. Training 15 min plus sampling and overhead 3 min is 0.3 GPU-hours per order, about $0.60 at $2 per GPU-hour. At 20,000 orders per day that is \( 20{,}000\times0.3=6{,}000 \) GPU-hours, or 250 GPUs if load were flat. With a peak-to-average ratio near 2 and jobs needing to start quickly, plan for 400 to 500 at peak. Storage per order is about 15 uploads x 3 MB plus 50 outputs x 1.5 MB, roughly 120 MB, so a 7 to 30 day retention keeps tens of TB live. Keeping full checkpoints would add 5 GB x 20,000 = 100 TB per day, which is why “discard after delivery” carries so much weight.

7. Deployment and monitoring

Safety and privacy. Match a live selfie against the upload cluster to stop impersonation, block public figures and minors, run a safety classifier on outputs, and add a watermark plus C2PA credentials. Encrypt at rest, never train shared models on customer photos without opt-in, and log deletions.

Monitoring. SLA attainment, rejection rate by reason and subgroup, identity score and regenerations per order, queue age, and safety hit rate.

Rollout. A base-model, template or hyperparameter change goes to a small share of orders first and is compared with the current version on identity score, regenerations per order, gate rejection rate by subgroup and refund requests. Each order trains its own model, so a rollback affects only new orders; orders in flight finish on the version they leased, which keeps results reproducible for support cases.

Feedback. Downloads per image and support tickets show which templates work, and templates with low download rates are rewritten or retired. Customer photos never enter shared training without opt-in, so feedback improves templates, thresholds and the validator rather than the base model. Recalibrate the identity threshold whenever the base or face recognition model changes, and page on-call when SLA attainment stays below target for an hour.

Follow-ups and extensions

• Results in seconds: a tuning-free encoder gives an instant preview; the finetuned set is the paid tier.
• Returning users: store a LoRA adapter with consent and ship new template packs.
• Group photos: identifiers bleed into each other; use separate adapters with regional prompting.
• Aspect ratios: SDXL’s size and crop conditioning handles 4:5 or 16:9.
• Forgetting: add an L2 penalty toward base weights, or default to LoRA.

Interview questions

Clarifying

Which clarifying answer changes the design most? Latency. An hour allows per-user finetuning and the highest-fidelity method. Seconds forces a tuning-free encoder and weaker identity. Next come photo count, model retention and user prompts, which decide DreamBooth versus LoRA and whether prompt filtering is needed.

Why does it matter whether the personalized model is stored? Full DreamBooth makes a 5 GB checkpoint per user. Deleted after delivery, it costs only GPU time. Kept, it becomes a large recurring storage bill, and adapters a hundred times smaller win. A stored model is also a biometric artifact that can generate the person forever, so retention changes the privacy story too.

What photos should users upload? Ten to twenty, chosen for variety rather than count. Different days, lighting, expressions and head angles, one face per photo, nothing covering it. Fewer photos overfit fast and cover too few angles. Showing good and bad examples at upload time is cheap and improves output more than most model changes.

Design and trade-offs

Tuning-free encoder or per-user finetuning? An encoder serves everyone from one model in seconds and costs little to run. It compresses identity into an embedding from one or a few photos, so details from other angles are lost, and training it needs large same-person paired data. Finetuning sees all the user’s photos and captures the face better, for 10 to 20 GPU-minutes per order. With an hour allowed and identity as the product, I finetune and keep the encoder for previews.

DreamBooth or LoRA, and when does the answer flip? Full DreamBooth gives the most detail, and its main cost disappears when weights are discarded. LoRA needs far less memory and produces megabyte adapters with close fidelity. The answer flips when users keep models, when budget pushes training onto 24 GB cards, or when many users must share one warm base. I’d settle it with an offline comparison on a held-out panel, since the gap depends on the base model.

Why a rare token instead of a name or random string? A real word has meaning the model must unlearn, and some of it leaks into images. A random string splits into several sub-tokens with their own associations. A rare single-unit token has a weak prior, so the face attaches to it cleanly. Adding the class noun lets the model reuse everything it knows about people.

What does prior preservation buy, and what does it cost? It keeps “a man” from collapsing onto the customer and preserves pose and background variety by training on the frozen model’s own class samples. It roughly doubles compute per step and needs a one-time class set. Too high a \( \lambda \) and the face stays soft; too low and variety and prompt control fade. I tune \( \lambda \) and step count together.

Deep dives

Why isn’t CLIP or DINO similarity enough for identity? CLIP encodes semantics, so two men in suits on gray backdrops score high. DINO responds to color and texture, catching hair or skin changes, but lighting and clothes still dominate. Neither was trained to separate people. A face recognition model trained with a margin loss keeps the same person close across pose and lighting, so it is the primary identity metric and the others are secondary.

How do you set the identity threshold in the gate? Collect generated-versus-real pairs labeled same or different person, ideally by the people themselves. Pick the threshold for a target false-accept rate, because delivering a stranger is worse than regenerating. Check rejection rates by subgroup, since face models are uneven across groups. Recalibrate whenever the base or recognition model changes.

How do you detect memorization of training photos? Compare each output to its nearest upload using both a face embedding and a full-image metric like LPIPS or DINO. Matching pose, background and expression almost exactly means a copy, even with a perfect identity score. Track the share of copies per order. A rise after a hyperparameter change points to too many steps or too little prior weight.

Estimate GPU memory for full DreamBooth on a 2.6B-parameter U-Net. Weights, gradients and two Adam moments in mixed precision come to about 16 bytes per parameter, so about 42 GB. Attention activations at 128x128 latents add tens of GB, so gradient checkpointing is needed on an 80 GB GPU. 8-bit Adam saves about 15 GB. LoRA needs roughly 5 GB of frozen weights plus activations and fits a 24 GB card.

Estimate the fleet for 20,000 orders a day. At 0.3 GPU-hours per order, that is 6,000 GPU-hours a day, or 250 GPUs at flat load. Peaks around twice the average and the need to start jobs quickly push it to 400 to 500 GPUs at peak with autoscaling. At $2 per GPU-hour compute is about $0.60 per order. LoRA training at 8 minutes would cut this by about a third.

Operations and failure modes

Two of eighteen photos show someone else. What happens? The model learns a blend, and every output looks subtly wrong. Embedding all faces and keeping the dominant cluster catches it before training. If the cluster is ambiguous, ask the user to confirm which photos are them. That check costs milliseconds; the failure costs a training run and a refund.

Regeneration rates jump after a base model upgrade. How do you debug? Split by template, subgroup and gate reason. If a few templates dominate, they render differently on the new base and need rewriting. If identity failures rise everywhere, the step count and \( \lambda \) tuned for the old base no longer fit, so rerun the sweep. Also confirm class latents were regenerated with the new base, because stale ones anchor to the wrong prior.

A user says the headshots look like someone else. What do you check? The validation record: accepted photo count, cluster tightness, pose spread. Then per-image identity scores and how many retries occurred. Typical causes are near-duplicate uploads, an outlier that slipped through, or a threshold too lenient for this user’s subgroup. Rerun with better photos for the user, and tighten the check that failed for the system.

A 45-minute walkthrough

Review questions

Answer each question in your own words first, then open the answer to check.

Concepts and definitions

1. Why can’t a well-written prompt alone produce headshots of a specific customer?

2. What are language drift and mode collapse in a personalized model?

3. Which parameters change per user in textual inversion, DreamBooth and LoRA?

4. Why are the prior-preservation class images generated by the frozen base model instead of collected from real photos?

5. Why is a face-recognition embedding the primary identity metric rather than CLIP or DINO features?

6. What does the per-image quality gate check before an output is delivered?

Mechanisms and math

7. How does the identifier token end up controlling what is drawn at each image location?

8. Why is LoRA’s \( B \) matrix initialized to zero, and why does LoRA add no inference cost?

9. What does each term of the DreamBooth loss do, and what does \( \lambda \) control?

10. In classifier-free guidance, what does \( w=1 \) mean, and why do personalized models usually run with a lower \( w \) than the base?

11. Why is FID used only as a system-level metric in this design?

Numbers and estimates

12. How many trainable parameters does rank-16 LoRA add to one 1280x1280 projection, and what fraction is that?

13. Estimate sampling time for one order of 50 images on an A100.

14. Estimate per-order storage and compare it with keeping full checkpoints at 20,000 orders per day.

Design reasoning and trade-offs

15. Why do training and sampling run on the same GPU worker?

16. Why autoscale the GPU fleet on queue age instead of GPU utilization?

17. What does a curated template library give the team compared with letting users type prompts?

18. Why cap regenerations at about three tries and then switch to a reserve template?

Common mistakes

19. What goes wrong if the same accessory, such as glasses or a red hoodie, appears in every uploaded photo?

20. What goes wrong if outputs are ranked by identity score alone?