Skip to main content
All Docs
FeaturesMaking Tax DigitalUpdated April 10, 2026

AgentOS Bulk Import: Batched Property Sync (PERF-17)

AgentOS Bulk Import: Batched Property Sync (PERF-17)

Released in v1.0.481 · Performance patch

Overview

The agentos-bulk-import background function synchronises properties, tenancies, and transactions from AgentOS into the Making Tax Digital platform. As of v1.0.481, the property sync step has been optimised to use batched database upserts, eliminating a sequential N+1 pattern that affected users with larger property portfolios.

Background

The agentos-bulk-import Inngest function runs several sequential steps when syncing data from AgentOS:

  1. import-properties — imports raw property records
  2. import-tenancies — imports tenancy records
  3. import-transactions — imports financial transactions
  4. sync-to-properties-table — syncs imported properties into the main properties table (the one the UI reads)

Steps 1–3 already used batched upserts. Step 4 previously issued one database call per property.

The Problem

For a landlord or estate agent with N properties, the sync-to-properties-table step previously executed N sequential INSERT ... ON CONFLICT DO UPDATE statements. At 50 properties this is marginal; for estate agents managing larger portfolios (100+), the cumulative latency is significant and inconsistent with the rest of the function.

The Fix

All property insert values are now pre-computed into a single array, then upserted in chunks of 50 (PROPERTY_SYNC_BATCH_SIZE) via the same chunk() helper used elsewhere in the function.

Round-trips: N  →  ⌈N / 50⌉

Example — 200 properties:
  Before: 200 round-trips
  After:    4 round-trips

Why batch size 50?

The properties table uses a partial unique index on (org_id, agentos_property_id) WHERE agentos_property_id IS NOT NULL. The COALESCE expressions in the conflict resolution clause add per-row overhead in the query planner. A batch size of 50 keeps individual query payloads reasonable while delivering the bulk of the round-trip savings.

Address Field Preservation

A key constraint during this refactor was preserving existing town and postcode values that a landlord may have manually corrected in the UI.

Previous approach (per-row only):

// Conditional spread — only included in SET if non-empty
...(town ? { town } : {}),
...(postcode ? { postcode } : {}),

This required individual inserts because each row could have a different SET clause.

New approach (batch-compatible):

town    = COALESCE(NULLIF(excluded.town, ''),    properties.town)
postcode = COALESCE(NULLIF(excluded.postcode, ''), properties.postcode)

NULLIF(excluded.town, '') returns NULL when the incoming value is an empty string. COALESCE then falls back to the existing stored value. The behaviour is identical to the old conditional spread — empty incoming values never overwrite existing data.

Affected Users

  • All users who have connected an AgentOS account and run property syncs
  • The improvement is most noticeable for estate agents or property managers syncing portfolios of 50+ properties
  • No action is required — the next scheduled or triggered sync will automatically use the new batched path

No Behavioural Changes

This is a pure performance optimisation. The set of properties written, the conflict resolution logic, and the preservation of manually-edited address fields are all unchanged.