senofy/autofirmer-expanded
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Old rules for older accounts, a rules date per account, and a firm cap

Browse Source 
When the master changes an account type with "New accounts only", it now
serves the type's old rules as kept rule sets (firm.keptAccountRules), each
for accounts dated inside [openedAt, closedAt). This build applies them:

- getAccountConfig, the one rules lookup trading reads, puts a kept set's
  rule fields over the live type when the account's rules date falls in
  its window. Id, label, prefix, size and the projection fields stay the
  live type's, so every report still names the live type.
- account_assignments gains rules_date and rules_date_manual. A new row is
  dated when the machine first sees the account, on the master's clock
  when this one runs behind and never before the catalog's last change.
  Auto dates come down to the broker's evidence: the end of the first fund
  day in Chicago, or a DXtrade registration instant. Evidence is applied
  before an account's first trade. Rows from before the migration get its
  time; unpinning keeps the row and its date. Backups carry both columns.
- An account on a kept set stays on its type while its prefix and cached
  size still match, so a tie-break flip can't move it off its old rules.
- Kept entries parse through the same normalizer as live types (now
  normalizeAccountType in lib/kept-rules.ts); a malformed one rejects the
  sync like a malformed type. The cached catalog records the parser that
  stored it, and a cache from an older parser is fetched in full rather
  than trusted with ?since= (this also brings in maxConcurrentAccounts).
- Reports add keptRuleIds per firm and live counts per kept set per type.
- The dashboard row and the account page show the rules date next to the
  account name, with an "old rules" chip saying what differs. Clicking
  the date edits it inline (PUT /api/accounts/:id/rules-date), previewing
  which rules the date lands on.

Also: the master's per-firm cap on accounts traded at once
(maxConcurrentAccounts), applied on top of the machine's own max
concurrent when a batch is drawn and in copy trading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Brandon Li
2026-10-18 22:21:55 -05:00
co-authored by Claude Opus 5.5
parent 98016b3eb3
commit c0899620d4
24 changed files with 2127 additions and 153 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/firm-catalog.ts
+152 -47
View File
@@ -19,8 +19,10 @@
import { isFirmPlatform } from './master-protocol';
import { sanitizeInstruments, sanitizeInstrumentLimits, DEFAULT_MAX_ENTRY_SPREAD } from './instrument-specs';
import { normalizeAccountType } from './kept-rules';
import type {
CatalogAccountType, CatalogFirm, CatalogTradingSettings, FirmCatalogPayload, FirmPlatform, SymbolMap, SymbolMapCandidate,
CatalogAccountType, CatalogFirm, CatalogTradingSettings, FirmCatalogPayload, FirmPlatform, KeptAccountRules, SymbolMap,
SymbolMapCandidate,
} from './master-protocol';
import {
getSetting,
@@ -39,24 +41,50 @@ const FETCH_TIMEOUT_MS = 10_000;
const COLD_RETRY_MIN_MS = 10_000;
const COLD_RETRY_MAX_MS = 60_000;
/** The version of validatePayload below, stamped on the payload it caches.
* Bump it whenever the parser starts keeping a field an older one dropped:
* a cache from before then lacks the field however often it is re-read, and
* `?since=` would answer "unchanged" until the master next changes, so the
* first sync on a newer parser fetches the whole catalog.
* 2: kept rule sets (keptAccountRules) and the firm's maxConcurrentAccounts.
* Stamped only on a payload from a master that sends keptAccountRules: one
* from an older master can't vouch for kept sets it never sent. */
export const CATALOG_PARSER_VERSION = 2;
/** A Date header further off than this is somebody else's clock, not the master's. */
const MAX_MASTER_OFFSET_MS = 365 * 24 * 60 * 60 * 1000;
interface CatalogCache {
payload: FirmCatalogPayload | null;
/** Index over every firm's account types, for the id lookup. */
typesById: Map<string, CatalogAccountType>;
/** Each live account type's kept rule sets, by closedAt (oldest window
* first). Kept ids are deliberately not in typesById: nothing can be
* pinned to one. */
keptByLiveId: Map<string, KeptAccountRules[]>;
}
const g = global as typeof globalThis & {
__firmCatalog?: CatalogCache;
__firmCatalogTimer?: ReturnType<typeof setTimeout>;
__firmCatalogColdDelay?: number;
/** The master's clock minus this one, from the last sync's Date header. */
__masterOffsetMs?: number;
};
function indexPayload(payload: FirmCatalogPayload | null): CatalogCache {
const typesById = new Map<string, CatalogAccountType>();
const keptByLiveId = new Map<string, KeptAccountRules[]>();
for (const firm of payload?.firms ?? []) {
for (const type of firm.accountTypes) typesById.set(type.id, type);
for (const kept of firm.keptAccountRules ?? []) {
const list = keptByLiveId.get(kept.keptFrom);
if (list) list.push(kept);
else keptByLiveId.set(kept.keptFrom, [kept]);
}
}
return { payload, typesById };
for (const list of keptByLiveId.values()) list.sort((a, b) => Date.parse(a.closedAt) - Date.parse(b.closedAt));
return { payload, typesById, keptByLiveId };
}
function cache(): CatalogCache {
@@ -66,7 +94,14 @@ function cache(): CatalogCache {
const raw = getSetting('catalog_payload');
if (raw) {
try {
payload = validatePayload(JSON.parse(raw));
const stored = JSON.parse(raw);
payload = validatePayload(stored);
// Whatever parsed the blob when it was stored, not this parser: a
// field an older build dropped on the way in can't come back from
// re-reading the blob.
const parser = (stored as { parser?: unknown }).parser;
if (typeof parser === 'number') payload.parser = parser;
else delete payload.parser;
} catch {
console.error('[catalog] cached payload is unreadable — ignoring it');
}
@@ -87,6 +122,37 @@ export function getAccountTypeById(typeId: string): CatalogAccountType | undefin
return cache().typesById.get(typeId);
}
/** The kept rule sets of one live account type, oldest window first ([] when
* it keeps none). lib/account-type-resolver.ts picks an account's by its
* rules date. */
export function getKeptRulesFor(liveTypeId: string): KeptAccountRules[] {
return cache().keptByLiveId.get(liveTypeId) ?? [];
}
/** The kept rule set ids loaded for a firm, for the report — null while the
* cached catalog came from a parser older than CATALOG_PARSER_VERSION, which
* can't be trusted to hold them (the master then reads this machine as not
* applying kept rules yet). */
export function getKeptRuleIdsForFirm(firmId: string): string[] | null {
const payload = cache().payload;
if (!payload || (payload.parser ?? 0) < CATALOG_PARSER_VERSION) return null;
return (payload.firms.find((f) => f.id === firmId)?.keptAccountRules ?? []).map((k) => k.id);
}
/** The master's clock minus this machine's, as of the last good sync (null
* before one this process). Dates new accounts on the master's clock when
* this one runs behind (lib/kept-rules.ts stampRulesDate). */
export function getMasterOffsetMs(): number | null {
return g.__masterOffsetMs ?? null;
}
/** When the cached catalog last changed on the master (its payload's
* updatedAt); null with no cache. No account first seen after a change can
* be dated before it. */
export function getCatalogUpdatedAt(): string | null {
return cache().payload?.updatedAt ?? null;
}
export function getCatalogMeta(): {
version: number;
updatedAt: string | null;
@@ -131,7 +197,10 @@ export function getTradableFirms(): (CatalogFirm & { username: string; password:
// ── Sync ─────────────────────────────────────────────────────────────────────
const g2 = global as typeof globalThis & { __catalogUnknownPlatformWarned?: Set<string> };
const g2 = global as typeof globalThis & {
__catalogUnknownPlatformWarned?: Set<string>;
__catalogKeptSkipWarned?: Set<string>;
};
/** Once per firm per process — the catalog is re-validated every 60s. */
function warnUnknownPlatform(id: string, name: string, platform: unknown): void {
@@ -164,12 +233,41 @@ function sanitizeSymbolMap(raw: unknown): SymbolMap {
return map;
}
function isStages(value: unknown): value is { profit: number; consistency: number; minTradingDays: number }[] {
return Array.isArray(value) && value.every((s) =>
typeof s === 'object' && s !== null
&& typeof (s as Record<string, unknown>).profit === 'number'
&& typeof (s as Record<string, unknown>).consistency === 'number'
&& typeof (s as Record<string, unknown>).minTradingDays === 'number');
function isDateString(value: unknown): value is string {
return typeof value === 'string' && Number.isFinite(Date.parse(value));
}
/**
* A firm's kept rule sets, each as strict as an account type: a malformed
* entry throws and the whole sync is rejected. One that names no live type
* of this firm is skipped instead (warned once) — no account can map to it.
*/
function parseKeptRules(
firm: { id: string; name: string },
raw: unknown[],
accountTypes: CatalogAccountType[],
tradesEveryDay: boolean,
): KeptAccountRules[] {
const liveIds = new Set(accountTypes.map((t) => t.id));
const kept: KeptAccountRules[] = [];
raw.forEach((k: unknown, j: number) => {
const where = `kept rules ${j} of ${firm.name}`;
const type = normalizeAccountType(k, { tradesEveryDay, where });
const entry = k as Record<string, unknown>;
if (typeof entry.keptFrom !== 'string') throw new Error(`${where} is missing keptFrom`);
if (entry.openedAt !== null && !isDateString(entry.openedAt)) throw new Error(`${where} has bad openedAt`);
if (!isDateString(entry.closedAt)) throw new Error(`${where} has bad closedAt`);
if (!liveIds.has(entry.keptFrom) || type.firmId !== firm.id) {
const warned = (g2.__catalogKeptSkipWarned ??= new Set());
if (!warned.has(type.id)) {
warned.add(type.id);
console.warn(`[catalog] skipping kept rules ${type.id} of "${firm.name}": it names no account type of this firm (keptFrom ${entry.keptFrom}, firmId ${type.firmId})`);
}
return;
}
kept.push({ ...type, keptFrom: entry.keptFrom, openedAt: entry.openedAt, closedAt: entry.closedAt });
});
return kept;
}
/** Rejects anything that isn't a complete catalog. A half-parsed payload would
@@ -204,39 +302,13 @@ function validatePayload(raw: unknown): FirmCatalogPayload {
const tradesEveryDay = dx && firm.tradesEveryDay === true;
if (!Array.isArray(firm.accountTypes)) throw new Error(`firm ${firm.name} is missing accountTypes`);
const accountTypes: CatalogAccountType[] = firm.accountTypes.map((t: unknown, j: number) => {
if (typeof t !== 'object' || t === null) throw new Error(`account type ${j} is not an object`);
const type = t as Record<string, unknown>;
for (const key of ['id', 'firmId', 'label', 'prefix', 'updatedAt']) {
if (typeof type[key] !== 'string') throw new Error(`account type ${j} is missing ${key}`);
}
for (const key of [
'accountSize', 'profitTarget', 'consistency', 'minDayPnL',
'minTradingDays', 'maxLoss', 'maxPositionSize',
]) {
if (typeof type[key] !== 'number') throw new Error(`account type ${j} is missing ${key}`);
}
if (!isStages(type.withdrawalStages)) throw new Error(`account type ${j} has bad withdrawalStages`);
// Lenient like platform: a master from before the split toggle
// doesn't send the field, and those account types all split. Same
// for CME holiday combining, where the old behavior was "off".
return {
...type,
splitEnabled: type.splitEnabled !== false,
// An every-day firm has no CME calendar to fold holidays by.
combineCmeHolidays: type.combineCmeHolidays === true && !tradesEveryDay,
// Lenient: a master from before the min-legs field means "no minimum".
minLegs: typeof type.minLegs === 'number' ? type.minLegs : 0,
// Lenient: the obfuscation stops are opt-in, so absent means off.
firstDayStop: type.firstDayStop === true,
minDayStop: type.minDayStop === true,
// Lenient: no hard daily limit, no notional cap.
dailyLossLimit: nonNegative(type.dailyLossLimit),
// The firm's calendar, stamped on every type so the cfg-based
// paths (projections, completion, the dashboard) all see it.
tradesEveryDay,
} as unknown as CatalogAccountType;
});
const accountTypes: CatalogAccountType[] = firm.accountTypes.map((t: unknown, j: number) =>
normalizeAccountType(t, { tradesEveryDay, where: `account type ${j}` }));
// Only when the master sent the list: absent = a master from before
// kept rules, [] = every account on the current rules.
const keptAccountRules = Array.isArray(firm.keptAccountRules)
? parseKeptRules({ id: firm.id, name: firm.name }, firm.keptAccountRules, accountTypes, tradesEveryDay)
: undefined;
firms.push({
id: firm.id,
@@ -248,6 +320,7 @@ function validatePayload(raw: unknown): FirmCatalogPayload {
...(platform !== 'dxtrade' ? { instrumentLimits: sanitizeInstrumentLimits(firm.instrumentLimits) } : {}),
bannedSymbols: Array.isArray(firm.bannedSymbols) ? (firm.bannedSymbols as string[]) : [],
accountTypes,
...(keptAccountRules ? { keptAccountRules } : {}),
serverUrl: typeof firm.serverUrl === 'string' ? firm.serverUrl : '',
loginDomain: typeof firm.loginDomain === 'string' ? firm.loginDomain : '',
commissionPct: dx ? nonNegative(firm.commissionPct) : 0,
@@ -255,12 +328,24 @@ function validatePayload(raw: unknown): FirmCatalogPayload {
symbolMap: sanitizeSymbolMap(firm.symbolMap),
targetHoldHours: nonNegative(firm.targetHoldHours),
financingPctPerDay: nonNegative(firm.financingPctPerDay),
// Lenient: absent or malformed = no firm cap.
maxConcurrentAccounts: Math.floor(nonNegative(firm.maxConcurrentAccounts)),
updatedAt: typeof firm.updatedAt === 'string' ? firm.updatedAt : new Date(0).toISOString(),
});
});
const trading = parseTradingSettings(body.trading);
return { version: body.version, updatedAt: body.updatedAt, firms, ...(trading ? { trading } : {}) };
// A kept-aware master sends keptAccountRules on every firm (even []). A
// payload without it is a master from before kept rules: cached without
// a parser version, so the cache isn't trusted to hold kept sets — every
// sync fetches in full (no `since`), and the first one after a kept-aware
// master is back brings them, though its catalog version never moved.
const keptAware = body.firms.every((f: unknown) =>
typeof f === 'object' && f !== null && Array.isArray((f as Record<string, unknown>).keptAccountRules));
return {
version: body.version, updatedAt: body.updatedAt, firms, ...(trading ? { trading } : {}),
...(keptAware ? { parser: CATALOG_PARSER_VERSION } : {}),
};
}
/** The master's trading settings; undefined when it sent none (or nothing
@@ -293,6 +378,17 @@ function recordError(message: string): void {
setSetting('catalog_last_sync_at', new Date().toISOString());
}
/** The master's clock off a response's Date header (whole seconds, read on
* arrival — close enough to date accounts by). An unreadable header, or one
* a year off, leaves the last offset standing. */
function recordMasterClock(header: string | null): void {
const at = header ? Date.parse(header) : NaN;
if (!Number.isFinite(at)) return;
const offset = at - Date.now();
if (Math.abs(offset) > MAX_MASTER_OFFSET_MS) return;
g.__masterOffsetMs = offset;
}
/** If the master's database was rebuilt, its firms come back with fresh ids and
* every credential row orphans. Match the survivors up by name (once) rather
* than making someone retype every password. */
@@ -323,14 +419,17 @@ export async function syncCatalog(): Promise<{ ok: boolean; changed: boolean; ve
// before anyone has got round to naming it.
// `since` only against the master the cache came from: two masters'
// version numbers can collide, and an 'unchanged' from the wrong one
// would keep the other's catalog.
// would keep the other's catalog. Nor for a cache an older parser wrote:
// it lacks what that parser dropped, and 'unchanged' would keep it so.
const hasCache = getCachedCatalog().length > 0;
const source = url.origin;
if (hasCache && getSetting('catalog_source_url') === source) url.searchParams.set('since', String(version));
const cacheCurrent = (cache().payload?.parser ?? 0) >= CATALOG_PARSER_VERSION;
if (hasCache && cacheCurrent && getSetting('catalog_source_url') === source) url.searchParams.set('since', String(version));
try {
const res = await fetch(url, { signal: AbortSignal.timeout(FETCH_TIMEOUT_MS) });
if (!res.ok) throw new Error(`master returned ${res.status}`);
recordMasterClock(res.headers.get('date'));
const body = await res.json();
if (body && typeof body === 'object' && (body as Record<string, unknown>).unchanged === true) {
@@ -353,13 +452,19 @@ export async function syncCatalog(): Promise<{ ok: boolean; changed: boolean; ve
}
// Same reasoning one level down: a firm losing every account type
// unassigns all of its accounts. Keep the previous types and flag it.
// unassigns all of its accounts. Keep the previous types and flag it,
// with the kept rule sets that went with them (whose keptFrom named
// types this payload lacks, so none of its own survived parsing). An
// empty kept list on its own is never suspicious: it is every account
// on the current rules.
const previous = new Map(getCachedCatalog().map((f) => [f.id, f]));
const suspicious: string[] = [];
for (const firm of payload.firms) {
const before = previous.get(firm.id);
if (firm.accountTypes.length === 0 && before && before.accountTypes.length > 0) {
firm.accountTypes = before.accountTypes;
if (before.keptAccountRules) firm.keptAccountRules = before.keptAccountRules;
else delete firm.keptAccountRules;
suspicious.push(firm.name);
}
}