parent
e7d12be4b8
commit
aaf569cb69
GPG Key ID:
BFDB4EF72D73C5F2
@ -1,170 +1,190 @@
|
||||
# themis
|
||||
|
||||
Discord App to allow EU4 players to take claims on regions and provinces.
|
||||
|
||||
## Setup
|
||||
|
||||
### Requirements
|
||||
|
||||
To develop:
|
||||
- [Go](https://go.dev/) version 1.19 or higher installed locally
|
||||
- `sqlite3` installed locally (already ships by default on most OSes)
|
||||
|
||||
To deploy:
|
||||
- Register for a [free account on Fly](https://fly.io)
|
||||
- An [AWS account](https://console.aws.amazon.com)
|
||||
|
||||
### Steps
|
||||
|
||||
To work with the core modules that simply interact with the database, or make
|
||||
sure that everything compiles correctly, you can run the following commands:
|
||||
[](https://github.com/golang-migrate/migrate/actions/workflows/ci.yaml?query=branch%3Amaster)
|
||||
[](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)
|
||||
[](https://coveralls.io/github/golang-migrate/migrate?branch=master)
|
||||
[](https://packagecloud.io/golang-migrate/migrate?filter=debs)
|
||||
[](https://hub.docker.com/r/migrate/migrate/)
|
||||
|
||||
[](https://github.com/golang-migrate/migrate/releases)
|
||||
[](https://goreportcard.com/report/github.com/golang-migrate/migrate/v4)
|
||||
|
||||
# migrate
|
||||
|
||||
__Database migrations written in Go. Use as [CLI](#cli-usage) or import as [library](#use-in-your-go-project).__
|
||||
|
||||
* Migrate reads migrations from [sources](#migration-sources)
|
||||
and applies them in correct order to a [database](#databases).
|
||||
* Drivers are "dumb", migrate glues everything together and makes sure the logic is bulletproof.
|
||||
(Keeps the drivers lightweight, too.)
|
||||
* Database drivers don't assume things or try to correct user input. When in doubt, fail.
|
||||
|
||||
Forked from [mattes/migrate](https://github.com/mattes/migrate)
|
||||
|
||||
## Databases
|
||||
|
||||
Database drivers run migrations. [Add a new database?](database/driver.go)
|
||||
|
||||
* [PostgreSQL](database/postgres)
|
||||
* [PGX v4](database/pgx)
|
||||
* [PGX v5](database/pgx/v5)
|
||||
* [Redshift](database/redshift)
|
||||
* [Ql](database/ql)
|
||||
* [Cassandra](database/cassandra)
|
||||
* [SQLite](database/sqlite)
|
||||
* [SQLite3](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
|
||||
* [SQLCipher](database/sqlcipher)
|
||||
* [MySQL/ MariaDB](database/mysql)
|
||||
* [Neo4j](database/neo4j)
|
||||
* [MongoDB](database/mongodb)
|
||||
* [CrateDB](database/crate) ([todo #170](https://github.com/mattes/migrate/issues/170))
|
||||
* [Shell](database/shell) ([todo #171](https://github.com/mattes/migrate/issues/171))
|
||||
* [Google Cloud Spanner](database/spanner)
|
||||
* [CockroachDB](database/cockroachdb)
|
||||
* [YugabyteDB](database/yugabytedb)
|
||||
* [ClickHouse](database/clickhouse)
|
||||
* [Firebird](database/firebird)
|
||||
* [MS SQL Server](database/sqlserver)
|
||||
|
||||
### Database URLs
|
||||
|
||||
Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: `dbdriver://username:password@host:port/dbname?param1=true¶m2=false`
|
||||
|
||||
Any [reserved URL characters](https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters) need to be escaped. Note, the `%` character also [needs to be escaped](https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_the_percent_character)
|
||||
|
||||
Explicitly, the following characters need to be escaped:
|
||||
`!`, `#`, `$`, `%`, `&`, `'`, `(`, `)`, `*`, `+`, `,`, `/`, `:`, `;`, `=`, `?`, `@`, `[`, `]`
|
||||
|
||||
It's easiest to always run the URL parts of your DB connection URL (e.g. username, password, etc) through an URL encoder. See the example Python snippets below:
|
||||
|
||||
```bash
|
||||
go test ./...
|
||||
go build -o ./bin/ ./cmd/...
|
||||
$ python3 -c 'import urllib.parse; print(urllib.parse.quote(input("String to encode: "), ""))'
|
||||
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
|
||||
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
|
||||
$ python2 -c 'import urllib; print urllib.quote(raw_input("String to encode: "), "")'
|
||||
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
|
||||
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
|
||||
$
|
||||
```
|
||||
|
||||
## Operations
|
||||
|
||||
This application is deployed via [Fly](https://fly.io) on a single virtual
|
||||
machine instance. It uses an embed SQLite database as its database engine, and
|
||||
uses [Litestream](https://litestream.io) to replicate the database to an S3
|
||||
bucket.
|
||||
|
||||
### Application Configuration
|
||||
|
||||
The deployment configurations can be found in the [fly.toml](/fly.toml) file and
|
||||
you can find more information on the configuration options [in the official Fly
|
||||
documentation](https://fly.io/docs/reference/configuration/).
|
||||
|
||||
The virtual image is based off of the [Dockerfile](/Dockerfile) which is a
|
||||
multi-stage build that builds the main application binary, downloads Litestream
|
||||
and packages everything on top of an Ubuntu 22.04 base.
|
||||
|
||||
### Entrypoint
|
||||
|
||||
The application is started using a [custom entrypoint shell script](/start.sh)
|
||||
that is in charge of first restoring the database file through Litestream and
|
||||
then starting the main application as a [child process of Litestream's
|
||||
replication process](https://litestream.io/reference/replicate/#arguments).
|
||||
## Migration Sources
|
||||
|
||||
It's a very simple script but is necessary since the application doesn't have a
|
||||
persistent volume to rely on and must rehydrate its database file after every
|
||||
deployment.
|
||||
Source drivers read migrations from local or remote sources. [Add a new source?](source/driver.go)
|
||||
|
||||
### Environment Variables
|
||||
* [Filesystem](source/file) - read from filesystem
|
||||
* [io/fs](source/iofs) - read from a Go [io/fs](https://pkg.go.dev/io/fs#FS)
|
||||
* [Go-Bindata](source/go_bindata) - read from embedded binary data ([jteeuwen/go-bindata](https://github.com/jteeuwen/go-bindata))
|
||||
* [pkger](source/pkger) - read from embedded binary data ([markbates/pkger](https://github.com/markbates/pkger))
|
||||
* [GitHub](source/github) - read from remote GitHub repositories
|
||||
* [GitHub Enterprise](source/github_ee) - read from remote GitHub Enterprise repositories
|
||||
* [Bitbucket](source/bitbucket) - read from remote Bitbucket repositories
|
||||
* [Gitlab](source/gitlab) - read from remote Gitlab repositories
|
||||
* [AWS S3](source/aws_s3) - read from Amazon Web Services S3
|
||||
* [Google Cloud Storage](source/google_cloud_storage) - read from Google Cloud Platform Storage
|
||||
|
||||
| Env Var | Defined At |
|
||||
| ----------------------- | --------------------- |
|
||||
| `DISCORD_TOKEN` | fly secret |
|
||||
| `DISCORD_APP_ID` | [fly.toml](/fly.toml) |
|
||||
| `DISCORD_GUILD_ID` | [fly.toml](fly.toml) |
|
||||
| `AWS_ACCESS_KEY_ID` | fly secret |
|
||||
| `AWS_SECRET_ACCESS_KEY` | fly secret |
|
||||
## CLI usage
|
||||
|
||||
## Local Development
|
||||
* Simple wrapper around this library.
|
||||
* Handles ctrl+c (SIGINT) gracefully.
|
||||
* No config search paths, no config files, no magic ENV var injections.
|
||||
|
||||
### Application Entrypoint
|
||||
__[CLI Documentation](cmd/migrate)__
|
||||
|
||||
- `./cmd/themis-server` This is the main application entrypoint
|
||||
- `./cmd/themis-repl` _coming soon_
|
||||
### Basic usage
|
||||
|
||||
### Core Functions
|
||||
|
||||
The core database functions can be developed and tested locally easily, just run
|
||||
`go test ./...` to test your changes locally.
|
||||
```bash
|
||||
$ migrate -source file://path/to/migrations -database postgres://localhost:5432/database up 2
|
||||
```
|
||||
|
||||
You can also load a test database easily by using the `init.sql` script. You can
|
||||
use the cli sqlite3 client or connect using whatever SQL editor you prefer that
|
||||
can open sqlite3 connections.
|
||||
### Docker usage
|
||||
|
||||
```bash
|
||||
sqlite3 local.db < migrations/init.sql
|
||||
sqlite3 local.db
|
||||
# interactive SQLite session
|
||||
$ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate
|
||||
-path=/migrations/ -database postgres://localhost:5432/database up 2
|
||||
```
|
||||
|
||||
### Discord Integration
|
||||
|
||||
This is a work in progress, but I am currently using a dedicated Discord server
|
||||
with this application already signed into it. You can contact me directly to be
|
||||
invited into that server. From there you only need to set the following
|
||||
environment variables before launching the `./cmd/themis-server` entrypoint:
|
||||
## Use in your Go project
|
||||
|
||||
* API is stable and frozen for this release (v3 & v4).
|
||||
* Uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies.
|
||||
* To help prevent database corruptions, it supports graceful stops via `GracefulStop chan bool`.
|
||||
* Bring your own logger.
|
||||
* Uses `io.Reader` streams internally for low memory overhead.
|
||||
* Thread-safe and no goroutine leaks.
|
||||
|
||||
__[Go Documentation](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)__
|
||||
|
||||
```go
|
||||
import (
|
||||
"github.com/golang-migrate/migrate/v4"
|
||||
_ "github.com/golang-migrate/migrate/v4/database/postgres"
|
||||
_ "github.com/golang-migrate/migrate/v4/source/github"
|
||||
)
|
||||
|
||||
func main() {
|
||||
m, err := migrate.New(
|
||||
"github://mattes:personal-access-token@mattes/migrate_test",
|
||||
"postgres://localhost:5432/database?sslmode=enable")
|
||||
m.Steps(2)
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
export DISCORD_APP_ID="1014881815921705030"
|
||||
export DISCORD_GUILD_ID="[test server id goes here]"
|
||||
Want to use an existing database client?
|
||||
|
||||
```go
|
||||
import (
|
||||
"database/sql"
|
||||
_ "github.com/lib/pq"
|
||||
"github.com/golang-migrate/migrate/v4"
|
||||
"github.com/golang-migrate/migrate/v4/database/postgres"
|
||||
_ "github.com/golang-migrate/migrate/v4/source/file"
|
||||
)
|
||||
|
||||
func main() {
|
||||
db, err := sql.Open("postgres", "postgres://localhost:5432/database?sslmode=enable")
|
||||
driver, err := postgres.WithInstance(db, &postgres.Config{})
|
||||
m, err := migrate.NewWithDatabaseInstance(
|
||||
"file:///migrations",
|
||||
"postgres", driver)
|
||||
m.Up() // or m.Step(2) if you want to explicitly set the number of migrations to run
|
||||
}
|
||||
```
|
||||
|
||||
### Litestream replication
|
||||
## Getting started
|
||||
|
||||
Litestream is _not a necessary component_ to run the `./cmd/themis-server`
|
||||
entrypoint and can be safely ignored when developing locally. Still, if you wish
|
||||
to try it out, you can find the Litestream commands used in production in the
|
||||
[start.sh](/start.sh) script. As per the Litestream docs, it should work fine
|
||||
with [Minio](https://min.io/) but I have not tested it yet nor are there any
|
||||
scripts provided to run it (yet).
|
||||
Go to [getting started](GETTING_STARTED.md)
|
||||
|
||||
## SQLite
|
||||
## Tutorials
|
||||
|
||||
### Importing From CSV
|
||||
* [CockroachDB](database/cockroachdb/TUTORIAL.md)
|
||||
* [PostgreSQL](database/postgres/TUTORIAL.md)
|
||||
|
||||
This is a neat feature built-in to SQLite. Using the source file at
|
||||
[data/eu4-provinces.csv](/data/eu4-provinces.csv) you can import the data
|
||||
directly into a SQLite database using the following command:
|
||||
(more tutorials to come)
|
||||
|
||||
## Migration files
|
||||
|
||||
Each migration has an up and down migration. [Why?](FAQ.md#why-two-separate-files-up-and-down-for-a-migration)
|
||||
|
||||
```bash
|
||||
$ sqlite3
|
||||
# ...
|
||||
sqlite> .mode csv
|
||||
sqlite> .import data/eu4-provinces.csv provinces
|
||||
sqlite> .schema provinces
|
||||
CREATE TABLE provinces(
|
||||
"ID" TEXT,
|
||||
"Name" TEXT,
|
||||
"Development" TEXT,
|
||||
"BT" TEXT,
|
||||
"BP" TEXT,
|
||||
"BM" TEXT,
|
||||
"Trade good" TEXT,
|
||||
"Trade node" TEXT,
|
||||
"Modifiers" TEXT,
|
||||
"Type" TEXT,
|
||||
"Continent" TEXT,
|
||||
"Superregion" TEXT,
|
||||
"Region" TEXT,
|
||||
"Area" TEXT
|
||||
);
|
||||
sqlite> select count(1) from provinces;
|
||||
3925
|
||||
1481574547_create_users_table.up.sql
|
||||
1481574547_create_users_table.down.sql
|
||||
```
|
||||
|
||||
### Creating an SQL Dump From Imported Data
|
||||
[Best practices: How to write migrations.](MIGRATIONS.md)
|
||||
|
||||
The init script at [migrations/init.sql](/migrations/init.sql) was initially
|
||||
created after importing the CSV data and running the following commands:
|
||||
## Versions
|
||||
|
||||
```bash
|
||||
# In the same SQlite session as last section
|
||||
sqlite> .output dump.sql
|
||||
sqlite> .dump
|
||||
```
|
||||
Version | Supported? | Import | Notes
|
||||
--------|------------|--------|------
|
||||
**master** | :white_check_mark: | `import "github.com/golang-migrate/migrate/v4"` | New features and bug fixes arrive here first |
|
||||
**v4** | :white_check_mark: | `import "github.com/golang-migrate/migrate/v4"` | Used for stable releases |
|
||||
**v3** | :x: | `import "github.com/golang-migrate/migrate"` (with package manager) or `import "gopkg.in/golang-migrate/migrate.v3"` (not recommended) | **DO NOT USE** - No longer supported |
|
||||
|
||||
Note: The column names were _edited manually_ to remove capital letters and
|
||||
spaces to make it easier to work with.
|
||||
## Development and Contributing
|
||||
|
||||
### Claims Schema
|
||||
Yes, please! [`Makefile`](Makefile) is your friend,
|
||||
read the [development guide](CONTRIBUTING.md).
|
||||
|
||||
```sql
|
||||
CREATE TABLE claim_types (
|
||||
claim_type TEXT PRIMARY KEY
|
||||
);
|
||||
Also have a look at the [FAQ](FAQ.md).
|
||||
|
||||
CREATE TABLE claims (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
player TEXT,
|
||||
claim_type TEXT,
|
||||
val TEXT,
|
||||
FOREIGN KEY(claim_type) REFERENCES claim_types(claim_type)
|
||||
);
|
||||
```
|
||||
---
|
||||
|
||||
Looking for alternatives? [https://awesome-go.com/#database](https://awesome-go.com/#database).
|
||||
|
||||
@ -0,0 +1 @@
|
||||
package themis
|
||||
@ -0,0 +1,3 @@
|
||||
DROP TABLE claims;
|
||||
DROP TABLE claim_types;
|
||||
DROP TABLE provices;
|
||||
@ -1,6 +0,0 @@
|
||||
-- Part of #11
|
||||
ALTER TABLE claims ADD COLUMN userid TEXT;
|
||||
UPDATE claims SET userid='212714834490294273' WHERE player = 'shinemperor';
|
||||
UPDATE claims SET userid='345340157333078016' WHERE player = 'wperron';
|
||||
UPDATE claims SET userid='203896675960487936' WHERE player = 'Gillfren';
|
||||
UPDATE claims SET userid='264861923814801408' WHERE player = 'B i r b';
|
||||
Loading…
Reference in new issue