As part of the Resiliency Hub workshop, a relatively new service by AWS:
- Great recommendation based on AWS well-architected framework
- Assess RTO & RPO of your app based on the resilience policy
- Also provides recommendations to make it compliant
Liked the interface to assess the resiliency of an application in AWS:
S3 Batch Operations (for large jobs) for higher restore throughput to significantly reduce restore completion time π
(e.g. restore 10M+ objects such as sensor data in a few hours for ML analysis)
Concept of a virtual cluster within an EKS cluster is an interesting deployment perspective π
Watching Peter's session virtually - Performance is the new mantra as a differentiator: 1. Nitro (4 generations now) - that's why the focus on building its own chip with lower latency, higher throughput 2. Also supports sustainability benefits
New announcements by Peter during AWS reInvent: 1. New Nitro V5 released 2. EC2 C7 gn instance powered by Nitro 3. HPC7G instance for EC2 powered by Graviton 3E processor
Key points from Peter's session: 1. Focus on EFA for HPC, SRD (Scalable Reliable Datagram) built by AWS to overcome single path network limitation of TCP to multipath using SRD 2. All new EBS io2 volumes next year will use SRD) for high performance with lower tail latency
Key points from Peter's session: 3. ENA Express launched using SRD for low latency 4. ElasticCache with ENA Express reduces latency with higher throughput (taking advantage of multi-path networking
Key annoucements from Adam's session:
π OpenSearch Analytics Service
π Amazon Redshift integration for Apache Spark
π Amazon Aurora zero-ETL integration with Amazon Redshift - launched for near real-time analytics and ML on transactional data
Key annoucements from Adam's session:
πAmazon DataZone for Data Governance
π Operational paginated report in Amazon QuickSight
π ML-powered forecasting with Q
Key announcements from Adam's session:
π Amazon GuardDuty for EKS protection (launched earlier)
π Container Runtime Threat Detection launched
Amazon co-founded Opensource Cybersecurity framework and launched Security Lake supporting OCF today π
That's a key differentiator from other cloud providers - 600+ instance types for compute in AWS π #awsreinvent2022
New instance launch using Graviton π
For HPC workload, new launches π
Expo is always inspirational π
Netflix Chaos Engineering π
βͺοΈ FIT (failure injection technology) - next evolution of chaos engineering
βͺοΈ Services get resilience tested using the emulation applying scenarios, scor and treatment - triggers fallback and then measure the impact
Netflix Chaos Engineering π
βͺοΈCRR - By adding headers to the request determine injection points
βͺοΈProvides ability to trigger failure for certain devices or a specific request
βͺοΈFIT UI to plan/execute chaos test cases via the interface
Netflix Chaos Engineering π
βͺοΈApply Canary strategy to Chaos Engineering
βͺοΈCanary + FIT to limit the experiment
βͺοΈ Monitoring analysis using the event stream infra in real-time (e.g. comparing Canary and baseline monitoring to find trends)
End-to-end Chaos Engineering (evolution from Chaos Monkey to the new approach using FIT) π
Build your own Chaos Engineering discipline using π
Amazon recently launched Amazon CloudWatch Internet Monitor providing insights, monitoring, suggestions based on user profile, geographic granularity, health notification using event bridge, etc.
End-to-end data products are an important aspect of building the scalable data strategy π
Amazon is leveraging Opensource solution like Apache Spark to extend their capabilities π
A new capability for document database - fully managed dbms is prime offering and value-add by AWS π
Sagemaker extending its capabilities with support for Geospatial MLπ
Great to attend the workshop on using AWS HPC Compute with AWS ParallelCluster (uses opensource technologies such as Slurm, Spack, Modules)
AWS Nitro Enclaves (AWS Proprietary for Confidential Computing) - launched in 2020, going to get more traction using trusted compute environments in the future, particularly for financial services.
Great to do a workshop for blockchain processing using AWS Nitro Enclaves:
β’ β’ β’
Missing some Tweet in this thread? You can try to
force a refresh
Read this thread to know about Lakehouse architecture pioneered by Databricks (keeping it an active thread to keep all the key architecture knowledge as simple read) π§΅
(image source: Databricks)
#Databricks #Lakehouse #DeltaLake
Databricks along with UC Berkley & Stanford published this whitepaper in 2021 to revolutionize the DWH with Lakehouse architecture -> "Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics" people.eecs.berkeley.edu/~matei/papers/β¦
Databricks as a pioneer for unifying the warehouse and data lake as a Lakehouse platform, eliminating the complexity with foundation as delta lake.
Attending Snowflake Summit 2023 Keynote Session virtually - will be sharing notes as part of this thread as we progress π§΅
Feel free to share your learnings as a reply - will be glad to see snapshots from the people, who can attend in-person.
#snowflake #SnowflakeSummit2023
Frank Slootman (CEO) started with AI and Financial Services - with mention to DTCC, Fiserv, and Fidelity clearly indicating to bring more Financial Services clients addressing their Security, Governance, and other challenges - like the analogy with Data Universeπ
Native Applications Framework - one of the key announcements by Frank - to build apps similar to Mobile Appstore