Troubleshooting Server Workloads

This guide helps you diagnose and resolve common issues when working with .

Structure - Each issue follows a Symptom → Diagnosis → Solution → Verification pattern to guide you through systematic troubleshooting.

Universal issues

These issues can affect any Server Workload integration, regardless of authentication method.

Agent Controller not running or disconnected

Symptom -

Requests timeout or bypass Aembit entirely
Application uses placeholder credentials without replacement
No activity in Aembit logs

Diagnosis -

Check Agent Controller service status:

Linux (systemd) -

systemctl status aembit-agent-controller
# Should show "active (running)"

Windows -

Get-Service "Aembit Agent Controller"
# Should show Status: Running

Docker/Kubernetes -

kubectl get pods -n aembit-system
# Agent Controller pod should show STATUS: Running

Verify Agent Controller status in Aembit console:

Navigate to Edge Components > Agent Controllers
Find your Agent Controller
Check Status: Should show “Connected” (green indicator)

Solution -

If Agent Controller has stopped:

# Linux
sudo systemctl start aembit-agent-controller

# Windows (PowerShell as Administrator)
Start-Service "Aembit Agent Controller"

# Kubernetes
kubectl rollout restart deployment/aembit-agent-controller -n aembit-system

If connection status shows “Disconnected”:

Check Agent Controller logs for registration or connectivity errors
Verify the Agent Controller status in the Aembit console
Check network connectivity to the target service endpoint

Verification -

Retry your application’s request. It should succeed. Check Agent Proxy logs for credential injection:

# Linux
sudo journalctl --namespace aembit_agent_proxy -f

# Look for:
# "Request intercepted for server_workload=your-workload-name"
# "Credentials injected successfully"

Network connectivity issues

Symptom -

Agent Proxy or application can’t reach target service or
Timeouts when attempting authentication
DNS resolution failures

Diagnosis -

Test connectivity to target service (example for Entra ID):

curl -I "https://login.microsoftonline.com"
# Should return HTTP 200 or 400 (confirms endpoint is reachable)

Check DNS resolution for target service:

nslookup login.microsoftonline.com  # Example for Entra ID
# Should resolve to Microsoft IP addresses

Check firewall rules:

Verify firewall allows outbound HTTPS (port 443) to target service domain
Check network security groups (cloud environments)
Check corporate firewall rules (on-premises)

Solution -

Configure firewall to allow outbound HTTPS traffic:

Add target service domains (for example, *.microsoftonline.com for Entra ID, api.github.com for GitHub)

If using a corporate HTTP proxy:

# Set proxy environment variables for Agent Controller
export HTTP_PROXY=http://proxy.company.com:8080
export HTTPS_PROXY=http://proxy.company.com:8080

# Restart Agent Controller to apply
sudo systemctl restart aembit-agent-controller

If DNS resolution fails:

Verify DNS server configuration in /etc/resolv.conf (Linux)
Add custom DNS servers if needed
Check that corporate DNS can resolve public domains

Verification -

Retry the curl command to the target service. It should succeed:

curl -I "https://target-service.com"  # HTTP 200 or 400 (reachable)

Then retry the authentication request from your application.

Agent Proxy not intercepting traffic

Symptom -

Application makes requests but continues using placeholder credentials
Aembit logs show no activity
Requests reach target service with placeholder values (visible in service logs)

Diagnosis -

Verify Agent Controller configuration for traffic interception:

# View Agent Controller configuration
cat /etc/aembit/agent-controller/config.yaml  # Linux
# Or: C:\Program Files\Aembit\Agent Controller\config.yaml  # Windows

# Verify Server Workload is listed in configuration

Check Agent Proxy logs for interception activity:

# Linux - view real-time logs
sudo journalctl --namespace aembit_agent_proxy -f

# Linux - search for credential-related entries
sudo journalctl --namespace aembit_agent_proxy | grep -i "intercept\|credential"

# Look for log entries like:
# "Request intercepted for server_workload=your-workload-name"
# "Credentials injected successfully"

Common causes and solutions -

1. Agent Controller using outdated configuration

Fix - Restart Agent Controller to reload configuration:

sudo systemctl restart aembit-agent-controller  # Linux
# Or restart service in Windows Services

2. Application not routing traffic through Agent Proxy

Diagnosis - Check application’s HTTP proxy environment variables:

echo $HTTP_PROXY
echo $HTTPS_PROXY
# Should point to Agent Proxy (typically http://localhost:8080)

Fix - Set proxy environment variables before starting application:

export HTTP_PROXY=http://localhost:8080
export HTTPS_PROXY=http://localhost:8080
your-application-start-command

3. Application using system trust store but Aembit CA not installed

Symptom - SSL certificate verification errors in application logs

Diagnosis - Check if Aembit CA certificate is in system trust store:

# Linux
ls /etc/pki/ca-trust/source/anchors/ | grep -i aembit

# macOS
security find-certificate -c "Aembit" /Library/Keychains/System.keychain

Fix - Install Aembit CA certificate. See TLS Decrypt configuration for detailed instructions.

Verification -

After applying fixes, verify Agent Proxy intercepts requests:

# Monitor Agent Proxy logs while running your application
sudo journalctl --namespace aembit_agent_proxy -f  # Linux

# Trigger authentication request from your application
# (Run your app or call token acquisition method)

# Expected log output:
# "Request intercepted for server_workload=your-workload-name"
# "Credentials injected successfully"
# "Response returned to application"

If you see these log entries, Agent Proxy is correctly intercepting requests.

TLS Decrypt configuration issues

Symptom -

SSL certificate verification errors in application logs
SSLError: certificate verify failed
CERT_UNTRUSTED errors

Diagnosis -

Determine if your Server Workload requires TLS Decrypt:

Most Server Workloads require TLS Decrypt for intercepting HTTPS traffic
Not required for plain HTTP traffic

Verify TLS Decrypt configuration in Aembit console:

Navigate to Deploy & Install > Advanced Options > TLS Decrypt
Verify you enabled TLS Decrypt for your Agent Controller
Verify Aembit generated the CA certificate

Verify you installed the CA certificate on your system:

# Linux - check system trust store
ls /etc/pki/ca-trust/source/anchors/ | grep -i aembit
# Or: ls /usr/local/share/ca-certificates/ | grep -i aembit

# macOS - check keychain
security find-certificate -c "Aembit" /Library/Keychains/System.keychain

# Windows - check certificate store
certutil -store Root | findstr Aembit

Solution -

Step 1 - Enable TLS Decrypt in Aembit console (if not already enabled):

Navigate to Deploy & Install > Advanced Options > TLS Decrypt
Click Enable TLS Decrypt
Download the generated CA certificate

Step 2 - Install CA certificate on your system:

Linux (CentOS/Red Hat Enterprise Linux) -

# Copy CA certificate to trust store
sudo cp aembit-ca.crt /etc/pki/ca-trust/source/anchors/

# Update trust store
sudo update-ca-trust

Linux (Ubuntu/Debian) -

# Copy CA certificate to trust store
sudo cp aembit-ca.crt /usr/local/share/ca-certificates/

# Update trust store
sudo update-ca-certificates

macOS -

# Add to system keychain
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain aembit-ca.crt

Windows (PowerShell as Administrator) -

# Import to Trusted Root Certification Authorities
Import-Certificate -FilePath "aembit-ca.crt" -CertStoreLocation Cert:\LocalMachine\Root

Step 3 - Restart application to use updated trust store.

Verification -

Retry the request that was failing with SSL errors. It should now succeed without certificate verification errors.

Check application logs - no more SSLError or CERT_UNTRUSTED messages.

OAuth-specific issues

These issues apply to Server Workloads using OAuth authentication (Entra ID, Salesforce, GitHub OAuth, etc.).

OAuth token request fails

Applies to - Entra ID, Salesforce, GitHub (OAuth mode), Okta (OAuth mode)

Symptom -

Token endpoint returns HTTP 400 Bad Request
Token endpoint returns HTTP 401 Unauthorized
Application logs show “invalid_client” or “unauthorized_client” errors

Diagnosis -

Check token endpoint configuration in Server Workload:

Navigate to Workloads > Server Workloads in Aembit console
Select your Server Workload
Verify Token Endpoint URL is correct:
- Entra ID: https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token
- Salesforce: https://{instance}.my.salesforce.com/services/oauth2/token
- GitHub: https://github.com/login/oauth/access_token

Check configuration:

Navigate to Access Policies > Credential Providers
Select the Credential Provider used by your
Verify Client ID matches the application registration in the OAuth provider
For Authorization Code flow, verify Client Secret is current

Check Agent Proxy logs for specific error:

sudo journalctl --namespace aembit_agent_proxy | grep -i "token\|oauth\|error"

# Look for errors like:
# "Token request failed: invalid_client"
# "OAuth provider returned 401"

Solution -

If token endpoint URL is incorrect:

Update Server Workload configuration with correct URL
Save changes
Restart Agent Controller: sudo systemctl restart aembit-agent-controller

If Client ID or Secret is incorrect:

Verify credentials in OAuth provider (for example, Entra ID App Registration)
Update Credential Provider with correct values
Save changes
Test integration

If using Entra ID and getting “invalid_client”:

Verify the application registration exists and isn’t deleted
Check that the Directory (tenant) ID in token endpoint matches your Entra ID tenant
Verify you granted API permissions (see Permission or Scope Errors)

Verification -

Retry the token request. It should return HTTP 200 with an access_token in the response:

# Check application logs for successful token acquisition
# Expected: "Successfully received access token"
# Response should contain: access_token, expires_in, token_type

Permission or scope errors

Applies to - Entra ID, Salesforce, GitHub (OAuth mode)

Symptom -

Token request succeeds (HTTP 200)
But API calls return HTTP 403 Forbidden
Error messages like “insufficient_permissions” or “access_denied”

Diagnosis -

Check granted scopes vs. required scopes:

Entra ID -

Log in to Azure Portal
Navigate to Azure Active Directory > App registrations
Select your application
Click API permissions
Review granted permissions - verify the list includes all required permissions
Check Status column - should show green checkmark (administrator consent granted)

Salesforce -

Log in to Salesforce
Navigate to Setup > Apps > App Manager
Find your connected app
Click View → Manage Consumer Details
Review Selected OAuth Scopes

GitHub -

Log in to GitHub
Navigate to Settings > Developer settings > GitHub Apps
Select your app
Review Permissions section
Verify you selected the required permissions

Check scope configuration in Aembit Server Workload:

Navigate to Workloads > Server Workloads
Select your Server Workload
Verify Scopes field contains the required scopes
Compare with API documentation for required scopes

Solution -

If permissions are missing in OAuth provider:

Add required permissions in the OAuth provider (Azure Portal, Salesforce, GitHub)
For Entra ID: Click Grant administrator consent after adding permissions
Test the integration again

If scopes are incorrect in Server Workload:

Update Scopes field in Server Workload configuration
Save changes
Restart Agent Controller to reload configuration
Retry the request

If using Entra ID .default scope:

Verify the target API application defines the permissions your app needs
If permissions are recently added, wait 5-10 minutes for Azure AD to propagate changes
Consider using specific scopes instead of .default for better visibility

Verification -

Retry authentication to the protected resource:

# API call should now return HTTP 200-299 (success)
# No more 403 Forbidden errors

Check OAuth provider logs (if available):

Entra ID: Azure Active Directory > Sign-in logs → Filter by Application ID → Verify successful sign-ins (Status: Success)
GitHub: Check app installation logs
Salesforce: Setup > Event Monitoring → Check API events

API key issues

These issues apply to Server Workloads using API Key authentication (Okta, Claude, OpenAI, etc.).

Invalid API key errors

Applies to - Okta, Claude, OpenAI, GitHub (API Key mode), Stripe, Box

Symptom -

API returns HTTP 401 Unauthorized
Error messages like “Invalid API key” or “Authentication failed”
Application logs show authentication errors

Diagnosis -

Verify API key in Credential Provider is current and valid:

Navigate to Access Policies > Credential Providers in Aembit console
Select the Credential Provider used by your Access Policy
Review the API key value (Aembit may mask this value)

Check if the target service expired or revoked the API key:

Okta -

Log in to Okta Admin Console
Navigate to Security > API > Tokens
Verify your token appears in the list with Status “Active”
Check expiration date

OpenAI/Claude -

Log in to provider dashboard
Navigate to API keys section
Verify key is active (not revoked)

GitHub -

Navigate to Settings > Developer settings > Personal access tokens
Verify token is active and has required scopes

Check Agent Proxy logs for specific error:

sudo journalctl --namespace aembit_agent_proxy | grep -i "api.key\|401\|unauthorized"

# Look for errors like:
# "API request returned 401 Unauthorized"
# "Invalid API key format"

Solution -

If the API key expired or the service revoked it:

Generate a new API key in the target service (Okta, OpenAI, GitHub, etc.)
Copy the new API key
Update Credential Provider in Aembit console with new key
Save changes
Test the integration

If API key format is incorrect:

Okta: Ensure format uses Single Sign-On Web Services (SSWS) like SSWS {token} (note the space after SSWS)
OpenAI: Ensure format is sk-... (starts with sk-)
Claude: Ensure format is sk-ant-... (starts with sk-ant-)
GitHub: Ensure format is ghp_... (classic) or github_pat_... (fine-grained)

If header injection isn’t working:

Verify Authentication Scheme in Server Workload configuration:
- Bearer: Authorization: Bearer {api_key}
- Header: Custom header name like X-API-Key: {api_key}
Check Header field matches what the service expects
Verify you set Authentication Method to “API Key” or “HTTP Authentication”

Verification -

Retry the API request. It should return HTTP 200-299 (success):

# Check application logs for successful API call
# Expected: HTTP 200 response with valid data
# No more 401 Unauthorized errors

Test with curl (for debugging):

# This won't go through Aembit, but verifies the API key itself works
curl -H "Authorization: Bearer YOUR_API_KEY" https://api.service.com/endpoint

# Should return valid response

Database connection issues

These issues apply to Server Workloads using database authentication (MySQL, Postgres, Redis, etc.).

Connection refused or timeout

Applies to - MySQL, PostgreSQL, Redis, Snowflake

Symptom -

Database connection fails with timeout
Connection refused errors
Can’t establish connection to database server

Diagnosis -

Check database server is running and accessible:

For cloud databases (AWS RDS, GCP Cloud SQL) -

# Test network connectivity
nc -zv database.example.com 3306  # MySQL
nc -zv database.example.com 5432  # PostgreSQL
nc -zv database.example.com 6379  # Redis

# Should show: Connection to database.example.com port XXXX succeeded

For local databases -

# Check if database service is running
systemctl status mysql      # MySQL
systemctl status postgresql # PostgreSQL
systemctl status redis      # Redis

Check firewall and security group rules:

AWS RDS -

Navigate to RDS console
Select your database instance
Click Connectivity & security tab
Review Security groups - verify the rules allow your application’s IP or security group
Verify Publicly accessible setting matches your network topology

GCP Cloud SQL -

Navigate to Cloud SQL console
Select your instance
Click Connections tab
Verify Authorized networks includes your application’s IP range

On-premises -

# Check firewall rules (Linux)
sudo iptables -L | grep 3306  # MySQL
sudo iptables -L | grep 5432  # PostgreSQL

Check Server Workload configuration:

Navigate to Workloads > Server Workloads in Aembit console
Verify Host matches database server hostname or IP
Verify Port is correct (3306 for MySQL, 5432 for Postgres, 6379 for Redis)

Solution -

If database service isn’t running:

# Start database service
sudo systemctl start mysql      # MySQL
sudo systemctl start postgresql # PostgreSQL
sudo systemctl start redis      # Redis

If security group blocks connection:

Add inbound rule allowing traffic from application’s IP or security group
For AWS RDS: Add rule for TCP port 3306 (MySQL) or 5432 (Postgres) or 6379 (Redis)
For on-premises: Update firewall rules to allow traffic

If using private network:

Verify you configured Virtual Private Network (VPN) or Virtual Private Cloud (VPC) peering
Check route tables allow traffic between application and database subnets
Test connectivity from application server: telnet database.example.com 3306

Verification -

Retry the database connection from your application. It should succeed:

# Test with database client
mysql -h database.example.com -u username -p  # MySQL (will prompt for password)
psql -h database.example.com -U username      # PostgreSQL

# Connection should establish without timeout

Then verify application can connect through Aembit.

Authentication failed

Applies to - MySQL, PostgreSQL, Snowflake

Symptom -

Connection reaches database but login fails
Access denied for user errors (MySQL)
password authentication failed errors (PostgreSQL)
Database connection timeout after authentication attempt

Diagnosis -

Check Credential Provider configuration in Aembit:

Navigate to Access Policies > Credential Providers
Select the Credential Provider for your database
Verify Username matches database user
Verify Password is correct
For AWS RDS with IAM authentication, verify IAM role and token generation

Check database user permissions:

MySQL -

-- Connect as database admin
mysql -u root -p

-- Check if user exists
SELECT User, Host FROM mysql.user WHERE User='your_username';

-- Check user permissions
SHOW GRANTS FOR 'your_username'@'%';

PostgreSQL -

-- Connect as database admin
psql -U postgres

-- Check if user exists
\du your_username

-- Check database access
\l

-- Verify user has CONNECT privilege
SELECT datname, datacl FROM pg_database WHERE datname='your_database';

Check authentication method in database configuration:

MySQL (/etc/mysql/mysql.conf.d/mysqld.cnf):

# Verify authentication plugin
default_authentication_plugin=mysql_native_password  # or caching_sha2_password

PostgreSQL (/var/lib/pgsql/data/pg_hba.conf):

# Verify connection allowed for your user
# Example:
host    all    your_username    0.0.0.0/0    md5

Solution -

If username or password is incorrect in Credential Provider:

Verify credentials by testing direct connection to database
Update Credential Provider with correct credentials
Save changes
Retry connection through Aembit

If database user doesn’t exist:

MySQL -

-- Create user
CREATE USER 'your_username'@'%' IDENTIFIED BY 'your_password';

-- Grant permissions
GRANT ALL PRIVILEGES ON your_database.* TO 'your_username'@'%';
FLUSH PRIVILEGES;

PostgreSQL -

-- Create user
CREATE USER your_username WITH PASSWORD 'your_password';

-- Grant permissions
GRANT ALL PRIVILEGES ON DATABASE your_database TO your_username;

If using AWS RDS IAM authentication:

Verify IAM policy allows rds-db:connect action

Verify you created the database user with IAM authentication:

CREATE USER your_username IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';

Verify you configured the Credential Provider for IAM authentication

Verification -

Retry database connection. It should succeed:

# Application logs should show successful connection
# Expected: "Database connection established"
# No more "Access denied" or "authentication failed" errors

Test query execution:

cursor.execute("SELECT 1")
result = cursor.fetchone()
print(result)  # Should print: (1,)

Next steps

If you’re still experiencing issues after following these troubleshooting steps:

Check service-specific guides: See Server Workload Guides for service-specific troubleshooting
Review architecture: See Architecture Patterns to understand expected data flow
Contact support: Provide Agent Controller logs and specific error messages for faster resolution

Architecture Patterns - Understanding data flow for each authentication method
Developer Integration Guide - SDK integration and testing
Server Workload Guides - Service-specific configuration
TLS Decrypt Configuration - Detailed TLS setup
Agent Controller - Understanding the Agent Controller

Troubleshooting Server Workloads

Universal issues

Agent Controller not running or disconnected

Network connectivity issues

Agent Proxy not intercepting traffic

TLS Decrypt configuration issues

OAuth-specific issues

OAuth token request fails

Permission or scope errors

API key issues

Invalid API key errors

Database connection issues

Connection refused or timeout

Authentication failed

Next steps

Related resources