Writing Jobs

Create a New Class

Sync jobs should be placed in the app/NexusSync/Jobs/ folder, and generally follow the format <Type>From<Source>To<Target> (e.g. AccountsFromCrmToPasswordsDatabase).

Each class follows this format:

<?php

namespace App\NexusSync\Jobs;

use Alberon\NexusSync\Jobs\SyncJob;
use Illuminate\Database\Eloquent\Builder;

class AccountsFromCrmToPasswordsDatabase extends SyncJob
{
    protected const NAME = 'Accounts from CRM to Passwords Database';

    public function run(): void
    {
        //...
    }
}

Write the Sync Logic

The logic may vary, but it typically goes like this:

  • Load all records from the source system, filtered by $this->recordId if set, and index them by ID
  • Load all records from the target system, filtered by $this->recordId if set, and index them by the source ID
  • Loop through all source records - for each one:
    • If it in the target array, update the target record then remove it from the target array
    • If it is not in the target array, create a new record
  • Loop through everything remaining in the target array and mark them as inactive/deleted

Most of the time this will be acomplished using Eloquent models, but the same is possible using an API.

Note: If there are too many records to load into memory, it may be necessary to adapt this - e.g. start a database transaction, mark all records as inactive/deleted, then restore the ones you want to keep. This will be less efficient, but it shouldn't run into memory limits.

Look at the Alberon Intranet or turn IT on Sync applications for examples.

(TODO: In the future we should see if we can encapsulate this logic in one or more classes to simplify the jobs.)

Logging Messages

The base class includes several methods for logging messages:

$this->debug($message);   // grey
$this->info($message);    // white
$this->success($message); // green
$this->warning($message); // amber
$this->error($message);   // red

Each of these are added to the log and displayed in an appropriate colour.

Debug messages are hidden by default and require "Debug Sync Jobs" permission to view.

If there is at least one error message, the job is marked as failed. If there is at least one warning message, it is marked as "Finished with warnings".

To log a variable for debugging, use:

$this->debug($message, $variable);

// e.g.
$this->debug('$x =', $x)

The variable is converted to text using either json_encode() (if possible) or var_dump().

Logging Changes

Changes should be logged using:

$this->changed($message);

Which is displayed the same as success() but it also increments the changes counter.

There are also some helpers for logging database changes when using Eloquent models...

Update/Create an Eloquent Record

If you are updating a model, instead of calling $model->save() or $model->update(), use this format:

$model->fill([ ... ]); // Or set each field individually
$this->save($model, 'description');

// e.g.
$account->fill([
    'name' => $source->name,
    // ...
]);

$this->save($account, "account '{$source->name}' ({$source->id})");

Use the same method for creating new database records:

$account = Account::make([ ... ]);
$this->save($account, "account '{$source->name}' ({$source->id})")

They can be combined to support creating/updating in the same code - for example:

/** @var \App\Models\Passwords\Account $passwordsAccount */
$passwordsAccount = $passwordsAccounts->pull($crmAccount->id) ?? new PasswordsAccount();
$passwordsAccount->fill($attributes);
$this->save($passwordsAccount, "account '{$crmAccount->name}' ({$crmAccount->id})");

This example will generate one of the following log entries:

  • No changes to account '<name>' (<id>) (debug)
  • Created account '<name>' (<id>) (success + increment change counter)
  • Updated account '<name>' (<id>) (success + increment change counter)

It will also log the old and new values of each field.

Delete an Eloquent Record

Similarly, there is a delete() helper that logs the deletion and the old field values:

$this->delete($model, $description);

// e.g.
$this->delete($account, "account '{$account->name}' ({$account->id})")

This will be logged as Deleted account '<name>' (<id>).

For soft-deleting models, there is also a forceDelete() helper in case it is required.

Syncing Many-Many Records

Rather than calling $model->relation()->sync($ids), call:

$this->sync($model->relation(), $ids, $description);

// e.g.
$this->sync($casesContact->accounts(), $accountIds, "accounts for '{$crmContact->full_name}'");

This will be logged as Updated accounts for '<name>' (<id>), with the number of added/updated/deleted records recorded and the values written to the debug log.

Logging HTTP Requests

If you need to make HTTP requests, there is a helper to log the requests:

$response = $this->http()->get($url);

See the Laravel HTTP client for details - just replace Http:: with $this->http().

You can create your own helper methods if that would make things easier:

private function kbApi(): PendingRequest
{
    return $this->http()
        ->withOptions(['base_uri' => config('custom.kb_api_uri')])
        ->withBasicAuth('sync', config('custom.kb_api_token'))
        ->acceptJson();
}

private function getPageList()
{
    /** @var array $pages */
    $pages = $this->kbApi()
        ->get('sync/pages/clients', ['crm_account_id' => $this->recordId])
        ->throw()
        ->object();

    $this->debug('Found ' . count($pages) . ' KB page(s) in Clients.');

    return collect($pages);
}

By default, a summary of the request (method, URL) and the response (code) are logged, but the request bodies are not logged. For example:

[11:48:43.854]  HTTP Request:   PATCH https://kb.alberon.co.uk/api/sync/pages/clients/562
[11:48:43.944]  HTTP Response:  200 OK

If the HTTP response indicates an error (4xx or 5xx response code), the response will be recorded in full.

You can change this (for debugging) by overriding these constants:

protected const LOG_HTTP_REQUESTS = true;
protected const LOG_HTTP_REQUEST_BODY = false;
protected const LOG_HTTP_RESPONSES = true;
protected const LOG_HTTP_RESPONSE_BODY_ON_SUCCESS = false;
protected const LOG_HTTP_RESPONSE_BODY_ON_FAILURE = true;

Logging Exceptions

Any exceptions thrown during the job will be automatically caught, logged and reported to Sentry by the base class, and the job will be marked as failed.

If you want to manually catch/log an exception, in order to recover and continue the sync, you can do this:

try {
    //...
} catch (Throwable $exception) {
    $this->exception($exception);   // Add to the sync log
    report($exception);             // Report to Sentry - optional
}

Tips for Writing Sync Jobs

  • It is safer to soft-delete than hard-delete, in case there is a bug in the sync
    • However, a boolean flag, such as active or deleted, may be simpler than using deleted_at
  • When making HTTP requests, ensure you either check the response code or call ->throw() to ensure it throws an exception on failure
    • The ->retries() method is also useful for handling temporary errors