feat: Add solver database

[bgins]

Summary

This pull request makes the following changes:

• Add solver database store implementation
• Add new remove deal, result, and match decision store methods
• Add new get results and get match decisions store methods
• Add store test coverage
• Add database store configs
• Remove store log writers
• Configure solver to use database in local development
• Update integration tests to use the database store
• Fix AddResourceOffer interface method param name typo
• Rework integration tests

This pull request implements a database store for the solver. We will use it to replace the memory store to persist resource offers, job offers, deals, results, and match decisions.

A database implementation also enables horizontal scaling and a shared store over solver instances.

Task/Issue reference

Closes: #445

Test plan

This pull request switches the solver over to the new database implementation. Start the stack, run a few jobs, examine the database entries, run the integration tests.

Switch to the memory store and check that it continues to work by running a few jobs (see Details below for configuring the store implementation)

We have also included a new test suite to check the memory and database store implementations. Run it with:

go test -tags="integration" -count 1 -v pkg/solver/store/store_test.go

Or run all of the solver integration tests:

./stack integration-tests-solver

The solver database integration tests must be run before starting the resource provider for the main integration tests. If the resource provider is running, it deletes the resource provider's offer as part of cleanup and the main integration tests fail.

We have reworked the CI integration tests:

• Separate solver and main test groups with solver and main tags
• Start the base services Docker compose instead of the full compose-up stack
• Start the solver direct from Go
• Start the resource provider direct from Go

We run the solver integration tests before starting the resource provider and the main integration tests after.

Details

The Solver store is configured with environment variables or CLI options. For example, we configure local development like this:

lilypad/stack

Lines 198 to 200 in 49e4413

	export STORE_TYPE=database
	export STORE_CONN_STR=postgres://postgres:postgres@localhost:5432/solver-db?sslmode=disable
	export STORE_GORM_LOG_LEVEL=silent

The STORE_TYPE can be database or memory. When the STORE_TYPE is database, the STORE_CONN_STR connection string must be set.

STORE_GORM_LOG_LEVEL must be silent, info, error, or warn. The Gorm log level is primarily intended for local development use cases, and we set it to silent by default.

The help text reports the configuration options like this:

--store-conn-str string            The database store connection string (STORE_CONN_STR).
--store-gorm-log-level string      The database store gorm log level, one of "silent", "info", "error", "warn" (STORE_GORM_LOG_LEVEL). (default "silent")
--store-type string                The store type used by the solver, one of "database" or "memory" (STORE_TYPE). (default "database")

All configurations are validated when running the solver command before starting the service.

Tasks

We have added three new interface methods to remove deals, match decisions, and results:

• New store methods
  • Add RemoveDeal method
  • Add RemoveResult method
  • Add RemoveMatchDecision method
  • Add GetResults method
  • Add GetMatchDecisions method

The new methods support our testing use case, but also fill out the store interface with methods we may use in the future.

The solver database implementation and test coverage tracked by method:

• Solver database implementation
  • Add AddJobOffer method
  • Add AddResourceOffer method
  • Add AddDeal method
  • Add AddResult method
  • Add AddMatchDecision method
  • Add GetJobOffers method
  • Add GetResourceOffers method
  • Add GetDeals method
  • Add GetDealsAll method
  • Add GetResults method
  • Add GetMatchDecisions method
  • Add GetJobOffer method
  • Add GetResourceOffer method
  • Add GetResourceOfferByAddress method
  • Add GetDeal method
  • Add GetResult method
  • Add GetMatchDecision method
  • Add UpdateJobOfferState method
  • Add UpdateResourceOfferState method
  • Add UpdateDealState method
  • Add UpdateDealMediator method
  • Add UpdateDealTransactionsJobCreator method
  • Add UpdateDealTransactionsResourceProvider method
  • Add UpdateDealTransactionsMediator method
  • Add RemoveJobOffer method
  • Add RemoveResourceOffer method
  • Add RemoveDeal method
  • Add RemoveResult method
  • Add RemoveMatchDecision method
• Test coverage
  • Add AddJobOffer coverage
  • Add AddResourceOffer coverage
  • Add AddDeal coverage
  • Add AddResult coverage
  • Add AddMatchDecision coverage
  • Add GetJobOffers coverage
  • Add GetResourceOffers coverage
  • Add GetDeals coverage
  • Add GetDealsAll coverage
  • Add GetResults coverage
  • Add GetMatchDecisions coverage
  • Add GetJobOffer coverage
  • Add GetResourceOffer coverage
  • Add GetResourceOfferByAddress coverage
  • Add GetDeal coverage
  • Add GetResult coverage
  • Add GetMatchDecision coverage
  • Add UpdateJobOfferState coverage
  • Add UpdateResourceOfferState coverage
  • Add UpdateDealState coverage
  • Add UpdateDealMediator coverage
  • Add UpdateDealTransactionsJobCreator coverage
  • Add UpdateDealTransactionsResourceProvider coverage
  • Add UpdateDealTransactionsMediator coverage
  • Add RemoveJobOffer coverage
  • Add RemoveResourceOffer coverage
  • Add RemoveDeal coverage
  • Add RemoveResult coverage
  • Add RemoveMatchDecision coverage

Related issues or PRs

Epic: https://github.com/Lilypad-Tech/internal/issues/340

[bgins]

Removed the jsonl implementation because it was only used by the solver store.

[bgins]

Removed the jsonl implementation because it was only used by the solver store.

[bgins]

Diff is slightly confusing here because we reordered the checks, but it's adding CheckStoreOptions.

[bgins]

Renamed this parameter, not a job offer.