dotfiles/.agents/skills/nixos-best-practices/rules/troubleshooting.md

title, impact, impactDescription, tags

title	impact	impactDescription	tags
Troubleshooting Configuration Issues	MEDIUM	Systematic debugging approach for configuration issues	debugging,errors,troubleshooting,verification

Why This Matters

Knowing how to systematically debug NixOS configurations saves hours of frustration and prevents making random changes hoping something works.

General Approach

Follow this order when debugging:

1. Read error messages completely - They usually tell you exactly what's wrong
2. Verify syntax - Check for missing brackets, quotes, commas
3. Validate config - Use nixos-rebuild test first
4. Check scope - Is overlay/module in correct location?
5. Trace dependencies - Are required inputs/imports present?

Common Error Patterns

"undefined variable 'inputs'"

Cause: inputs not passed via specialArgs.

Fix:

# flake.nix
outputs = { self, nixpkgs, ... }@inputs:
{
  nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
    specialArgs = { inherit inputs; };  # Add this
    modules = [ ./hosts/myhost ];
  };
};

"attribute 'package-name' not found"

Cause: Package doesn't exist, or overlay not applied.

Debug:

# Check if package exists in nixpkgs
nix search nixpkgs package-name

# Check if overlay is defined
nix eval .#nixosConfigurations.myhost.config.nixpkgs.overlays

# Verify package in your config's pkgs
nix repl
nix-repl> :l flake.nix
nix-repl> myhost.config.environment.systemPackages

Fix:

• If overlay issue: Move to correct location (see overlay-scope.md)
• If missing package: Use correct package name or add overlay

"error: The option 'some.option' does not exist"

Cause: Typo in option name, or option not available in your nixpkgs version.

Debug:

# Search for option
nixos-options | grep some-option

# Check option in current config
nix eval .#nixosConfigurations.myhost.config.some.option

Fix:

• Check option name spelling
• Verify nixpkgs version has this option
• Read option documentation for correct usage

"infinite recursion"

Cause: Circular dependency in config.

Debug:

# Use --show-trace to see where
nixos-rebuild build --flake .#hostname --show-trace

Fix:

• Look for mutual dependencies
• Use mkBefore/mkAfter for ordering
• Break circular reference

"hash mismatch" in flake.lock

Cause: Local changes or outdated lock file.

Fix:

# Update flake lock
nix flake update

# Or update specific input
nix flake lock update-input nixpkgs

Overlay Not Applied

Symptom: Package from overlay not found.

Debug steps:

1. Check overlay definition:

nix eval .#nixosConfigurations.myhost.config.nixpkgs.overlays

2. Check overlay location:

# If useGlobalPkgs = true, overlay should be here:
grep -r "nixpkgs.overlays" hosts/myhost/default.nix

# NOT here:
grep -r "nixpkgs.overlays" home-manager/home.nix

3. Verify overlay input:

nix flake metadata
# Look for your overlay input

Fix: See overlay-scope.md

Configuration Changes Not Applying

Symptom: Edit config, rebuild, but nothing changes.

Debug steps:

1. Verify rebuild ran successfully:

sudo nixos-rebuild switch --flake .#hostname
# Look for "success" message

2. Check current generation:

sudo nix-env --list-generations --profile /nix/var/nix/profiles/system

3. Verify new generation is active:

sudo nixos-version
# Should show timestamp of recent rebuild

4. Check if option actually changed:

nix eval .#nixosConfigurations.myhost.config.some.option

Fix:

• Ensure rebuild succeeded
• Check if service needs restart: systemctl restart service-name
• For Home Manager: home-manager switch (or sudo nixos-rebuild switch if integrated)

Build Fails with "store collision"

Symptom: Multiple packages trying to write to same path.

Cause: Usually duplicate package declarations or conflicting modules.

Fix:

• Remove duplicate package declarations
• Check for conflicting modules
• Use --show-trace to identify conflicting packages

Verification Commands

Check configuration without building

# Check syntax
nix flake check

# Dry run build
nixos-rebuild build --flake .#hostname --dry-run

# Evaluate specific option
nix eval .#nixosConfigurations.myhost.config.environment.systemPackages

Test configuration safely

# Build without activating
nixos-rebuild build --flake .#hostname

# Test configuration (rollback on reboot)
nixos-rebuild test --flake .#hostname

# Switch (persistent)
nixos-rebuild switch --flake .#hostname

Compare generations

# List system generations
sudo nix-env --list-generations --profile /nix/var/nix/profiles/system

# List Home Manager generations
home-manager generations

# Compare configurations
sudo nix diff-closures /nix/var/nix/profiles/system-*-link

Useful Tools

nix repl

nix repl
nix-repl> :l flake.nix                    # Load flake
nix-repl> myhost.config.environment.systemPackages  # Access config
nix-repl> :b myhost.config.system.build.toplevel   # Build

nixos-version

# Check current system version
nixos-version

# Should show something like:
# 25.05.20250123.abc1234

journalctl

# Check service logs for failures
sudo journalctl -xe

# Check NixOS rebuild logs
sudo journalctl -u nixos-rebuild

Getting Help

When asking for help, provide:

1. Full error message (not just "it doesn't work")
2. Minimal reproducible example
3. NixOS version: nixos-version
4. Relevant config files
5. What you've tried
6. Expected vs actual behavior

Useful commands:

# Show system info
nixos-version
uname -a

# Show flake info
nix flake metadata

# Export config for sharing
nixos-rebuild build --flake .#hostname
nix-store -qR result > system-closure

Common Debugging Workflows

Package not found

# 1. Search for package
nix search nixpkgs package-name

# 2. Check if it's available for your system
nix eval nixpkgs#legacyPackages.x86_64-linux.package-name

# 3. Check your overlays
nix eval .#nixosConfigurations.myhost.config.nixpkgs.overlays

# 4. Check if overlay provides it
nix eval .#packages.x86_64-linux.package-name

Overlay not applying

# 1. Verify overlay is defined
grep -r "overlays" hosts/myhost/

# 2. Check useGlobalPkgs setting
grep "useGlobalPkgs" hosts/myhost/default.nix

# 3. Evaluate overlay
nix eval .#nixosConfigurations.myhost.config.nixpkgs.overlays

# 4. Test package directly
nix build .#packages.x86_64-linux.package-name

Service not starting

# 1. Check service status
systemctl status service-name

# 2. View service logs
journalctl -xeu service-name

# 3. Check service config
nixos-rebuild build --flake .#hostname --show-trace

# 4. Test service manually
/path/to/service-binary --verbose