Creating a Data View (Manually)
Storing data logic in the database can simplify sharing your analytics
Creating an analytic query (whether AI or SQL) is an iterative process of query, review and test, revise, and query again. Once a complex analytic is complete, saving it as a database view allows you and others a simplified path to replicating the same logic with a single SELECT.
Let's Get Started!
To be able to validate your script, we'd ask that you generate a new file called Using_API_Keys.sh that you can commit to your branch of the SXT-Community/SXTAccreditation repo at the end of the activity.
Creating a Standard View
The first step in saving logic to the database as a view, is generating that database logic!
Get Creative!
We'll provide you a query below, however using this logic is not a requirement - in fact, we'd recommend you take some time to explore the data, and see what kind of interesting analytics you can discover!
Create the DB Logic
Standard views are nothing more than a SQL statement saved to the Space and Time Database and given an object name, so that it can be queried as if it were a table. As an example, let's start with the SQL statement below (or substitute your own!):
/* AAVE v3: Flash Loan value of wETH or wstETH pool, rolling 90 days
runtime: ≈5sec */
SELECT cast(time_stamp as date) as Loan_Date
, case when substr(Asset,1,6)='0xc02a' then 'wETH' else 'wstETH' end as Asset_Name
, sum(cast(amount as decimal(38,2)))/1e18 as Amt_Total
from AAVE_V3_ETHEREUM.POOL_EVT_FLASHLOAN
where time_stamp between current_date-91 and current_Date-1
and asset in( lower('0x7f39c581f595b53c5cb19bd0b3f8da6c935e2ca0') --wstETH
,lower('0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2')) --wETH
group by cast(time_stamp as date), asset
order by 1
Like any database object, you will need a public key to assign the object when saving to the database, to allow the view owner (holder of the private key) to authorize activities like CREATE, DROP, and creating biscuits.
Create the Keypair
If you've completed the "Create Your Own Table" accreditation section, the rest of this section will be a repeat.
Click here if you need a reminder on creating keypairs for database objects.
All resources (tables, views, etc.) on the Space and Time network must be created with a public/private keypair. The public key is added to the CREATE TABLE statement and saved on the network. The private key is used to sign self-describing authorization keys (aka biscuits) which authorize any request to that table.
This allows authorization to be decentralized, while allowing the owner of the table (aka the person holding the private key) to define and share signed biscuits that the table can verify as authentic. The self-describing nature of biscuits also allow that authorization to be limited to a particular table, person, time period, or subscription.
To get started quickly, you can use the SXTCLI to generate a new table keypair AND an admin-level biscuit, all with one command: Open up a terminal window and enter the command:
sxtcli biscuit generate wildcard
The above command will return:
- An ED25519 keypair (hex encoded) that can be used for creating tables, views, etc.
- A description of the biscuit capabilities, in this case:
- for ANY resource ("*") with a matching public key,
- allow ANY command ("*") - hence the name "wildcard" or admin biscuit
- The actual biscuit key itself
This output will be displayed as:
Biscuit KeyPair (hex)
Private key: D4EAC895937773E3047FE2E9AFA7004B80601BA0F687511BC23658D366FFB51B
Public key: 51297647296389ADABBB6E62230203C620DF47BF8DCAEEF1B1EC35507FBA48E0
Biscuit content:
Facts:
sxt:capability("*","*");
Biscuit:
EpABCiYKDnN4dDpjYXBhYmlsaXR5CgEqGAMiDwoNCIAIEgMYgQgSAxiBCBIkCAASIJK...
Note, the below key was modified to be invalid -- please generate your own fresh key!
Both Users and Tables / Views use the ED25519 keypair for security. However, there is a difference in encoding between the two:
- User authentication will expect Base64 encoding
- Tables and Views, which need it embedded in SQL, will expect Hex encoding
The CLI will return the correct encoding when generating new keys.
Store the private key somewhere safe! If you lose the private key, you lose control of the table, view, or resource!
Multiple tables or views can share a single keypair - for example, a User
table and a User_Settings
table may share the same keypair (and so wildcard biscuit), since they are related and often operated on together.
That said, it is NOT recommended to reuse keys too widely, as it broadens the exposure of any stolen or lost keys.
Congrats! You have created a new resource public/private keypair!
Combine Logic and Keypair
Let's put the query together with the keypair, and create the view!
CREATE VIEW SXTTemp.vw_Aave_wETH_wstETH_Pool_<My UserID>
with "public_key=51297647296..., access_type=public_read"
as
---
SELECT cast(time_stamp as date) as Loan_Date
, case when substr(Asset,1,6)='0xc02a' then 'wETH' else 'wstETH' end as Asset_Name
, sum(cast(amount as decimal(38,2)))/1e18 as Amt_Total
from AAVE_V3_ETHEREUM.POOL_EVT_FLASHLOAN
where time_stamp between current_date-91 and current_Date-1
and asset in( lower('0x7f39c581f595b53c5cb19bd0b3f8da6c935e2ca0') --wstETH
,lower('0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2')) --wETH
group by cast(time_stamp as date), asset
order by 1
The "with" command takes two arguments:
- public_key - the required hex formatted public key, to allow authorization.
- access_type - similar to tables, this specifies the default access allowed without a biscuit. We are setting
public_read
here, to allow anyone to run our analysis of this public dataset.
Note that for standard views, the user must also have access to the underlying tables as well as the view.
Click here to read more on the different `access_type` options available.
Access type defines the global default access type for the table or view. For example, you can make the table globally public or private. To set the default table permission, you add the property access_type=
to the WITH
statement during table creation. If omitted, SxT will default to "Permissioned" (aka Private) which is the most restrictive default permission.
-
PERMISSIONED (default setting) - No access is provided by default, and every request must include a biscuit authorization. If not set, your table will default to a Permissioned table.
- Syntax: modify your
WITH
statement to:
WITH "public_key=$RESOURCE_PUBLIC_KEY, access_type=PERMISSIONED"
(or omitaccess_type
) - Use-Cases: private or internal organizational data, such as CRM data, private application logs, organizational master data, user analytics, etc.
- Syntax: modify your
-
PUBLIC_READ - Everyone on the network will have
SELECT
access by default, allowing anyone on the network to run queries on the data. For example, all SxT Blockchain tables are PUBLIC_READ.- Syntax: modify your
WITH
statement to:
WITH "public_key=$RESOURCE_PUBLIC_KEY, access_type=PUBLIC_READ"
- Use-Cases: publicly accessible data like weather data, stock or blockchain data, social feeds, test data, or anonymized data for educational or hackathon purposes.
- Syntax: modify your
-
PUBLIC_APPEND - Everyone on the network will have
SELECT,INSERT
permissions by default, allowing anyone on the network to query OR insert new data. Only users with the appropriate biscuit authorization can update or delete data, or modify the table's structure. This is sometimes used as a public (append-only) logging mechanism, although it's important to note that anyone on the network can add any number of new records.- Syntax: modify your
WITH
statement to:
WITH "public_key=$RESOURCE_PUBLIC_KEY, access_type=PUBLIC_APPEND"
- Use-Cases: Rare, but some publicly accessible logs or anonymized data collection points.
- Syntax: modify your
-
PUBLIC_WRITE - Everyone on the network will have
SELECT,INSERT,UPDATE,DELETE
permissions by default, allowing anyone on the network to query, add, delete, or alter any data in the table. Only users with the appropriate biscuit authorization can modify the table or view's structure. This is often used by Space and Time in demo projects, as it allows easy and open access to try SxT features, without needing to worry about biscuits. Because literally anyone can do anything to the data, this is not typically used outside of demos.- Syntax: modify your
WITH
statement to:
WITH "public_key=$RESOURCE_PUBLIC_KEY, access_type=PUBLIC_WRITE"
- Use-Cases: Publically accessible temporary data caches (insert / process / delete), or test-dev tables , or for simplified demos of data processes (where data quality doesn't matter)
- Syntax: modify your
Executing the above SQL in the method of your choice (SXTCLI, API, or Space and Time Studio will compile and save the view.
Congrats! You've created your first standard view!
Here's another example of a good view analytic...
/* display 5 years of my own wallet TXN history
runtime: ≈4 seconds
probably a good candidate for a standard view... */
---
Select
substr(time_stamp, 1, 7) as YrMth
, sum(case
when from_address = my_Wallet
then value_/1e18 end) as Amt_Out
, sum(case
when to_address = my_Wallet
then value_/1e18 end) as Amt_In
, count(*) as Txn_Count
from ETHEREUM.TRANSACTIONS
join (select lower('0x123abd...') as my_Wallet)
on from_address = my_Wallet
or to_address = my_Wallet
where time_stamp between current_date-(5*365) and current_date-1
Group by 1 order by 1
Materialized View
If your analytic takes a significant amount of time to run, you may want the view to execute and cache the results, so the materialized view runs instantly! The cache will refresh automatically on a timer, by default, once a day.
As an example, let's change our above example to a materialized view that runs not against 91 days, but against the entire history of Aave V3, going back to March of 2022. This still only runs in about 5-6 seconds, probably not worth materializing, but for this example:
CREATE MATERIALIZED VIEW SXTTemp.mvw_Aave_wETH_wstETH_Pool_<My UserID>
with "public_key=51297647296..., access_type=public_read, refresh_interval=1440"
as
---
SELECT cast(time_stamp as date) as Loan_Date
, case when substr(Asset,1,6)='0xc02a' then 'wETH' else 'wstETH' end as Asset_Name
, sum(cast(amount as decimal(38,2)))/1e18 as Amt_Total
from AAVE_V3_ETHEREUM.POOL_EVT_FLASHLOAN
where time_stamp between '2022-04-01' and current_Date-1
and asset in( lower('0x7f39c581f595b53c5cb19bd0b3f8da6c935e2ca0') --wstETH
,lower('0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2')) --wETH
group by cast(time_stamp as date), asset
order by 1
We only had to make 2 changes from above:
- First 3 words of the command are now
CREATE MATERIALIZED VIEW
instead of justCREATE VIEW
. - Second line, added
refresh_interval
to thewith
statement. The number (1440) is the number of minutes between data refreshes.
We also changed the name slightly, to begin with SXTTemp.mvw_
to indicate "materialized view", however that is not a requirement, simply a naming convention.
Currently the maximum allowed refresh_interval is 1440, aka 24 hours. However, you can manually refresh at any time by calling the API: view-materialized-refresh-force
Executing the above SQL in the method of your choice (SXTCLI, API, or Space and Time Studio will compile and save the view.
Congrats! You've created your first materialized view!
You may recall from the "Data and Queries" activity, that the Space and Time Studio will automatically create materialized views for you! There are some limitations, for instance it will always create the materialized view in one specific schema.
Building views yourself allows you to place the view where you want, and to assign the prefered access_type
to restrict access, which allows you to enforce the use of biscuits.
Get Credit:
You'll get credit for this activity if you:
- Create a view
- Create a materialized view
- Be Creative! It might win you prizes (hint hint)
- Commit your work file to the SXT-Community/SXTAccreditation repo
How should I commit my branch changes?
When you're complete with the activity, simply commit your changes to your branch of work, then push to the SXT-Community/SXTAccreditation repo. Many IDEs have automated processes to manage this for you, but to do this manually by opening a temrinal / WSL window, and typing:
If you get an error on line 5 saying, The current branch stephens_work has no upstream branch
, this is because Github doesn't know about your branch yet. Simply run the command the git tool recommends - it will look something like:
git push --set-upstream origin <your_branch>
When you're done for the day, head over to SXT-Community/SXTAccreditation/Pulls page on Github, and (if you're logged in) you'll see a message like:
Click on the button above to create a Pull Request, give us some feedback in the description, and click "Create Pull Request". Done!
Updated 2 months ago