diff --git a/Makefile b/Makefile index c5d781b..fd47f9c 100644 --- a/Makefile +++ b/Makefile @@ -5,4 +5,7 @@ linter-image: docker build --rm -f Dockerfile.mdlinter -t danielguo/mdlinter . md-linter: - docker run -v ${CURR_DIR}:/workdir danielguo/mdlinter \ No newline at end of file + docker run -v ${CURR_DIR}:/workdir danielguo/mdlinter + +toc: + python3 generate_toc.py \ No newline at end of file diff --git a/README.md b/README.md index f5959f6..0cf8f06 100644 --- a/README.md +++ b/README.md @@ -6,92 +6,163 @@ This repo contains the notes/tutorials from my personal tech exploration. ## Table of contents -- AI/ML related - - [Build simple loal RAG](ai/local-rag/notebook.ipynb) - - [Build simple local agent](ai/ai-agent/notebook.ipynb) - - [AI Inference Concepts](ai/ai-inference-concpets/readme.md) -- [CKA related](https://github.com/danniel1205/tech-notes/tree/master/cka) -- K8S related - - [admission controllers](k8s/explore-admission-controllers/explore-admission-controllers.md) - - [antrea](k8s/explore-antrea/explore-antrea.md) - - [capd](k8s/explore-capd/explore-capd.md) - - [cert-manager](k8s/explore-cert-manager/explore-cert-manager.md) - - [etcd](k8s/explore-etcd/readme.md) - - [carvel-tool](k8s/explore-k14s/readme.md) & [kapp-controller](k8s/explore-kapp-controller/readme.md) - - [kubevirt](k8s/explore-kubevirt/explore-kubevirt.md) - - [kudo](k8s/explore-kudo/explore-kudo.md) - - [auth-in-k8s](k8s/explore-pinniped/auth-in-k8s.md) - - [rancher](k8s/explore-rancher/explore-rancher.md) - - [secrets store csi](k8s/explore-secrets-store-csi/explore-secrets-store-csi.md) - - [statefulset on vsphere](k8s/explore-statefulset-on-vsphere/explore-statefulset-on-vsphere.md) - - [how pod is created with network configured](k8s/how-pod-created-with-network-configured/readme.md) - - [kustomize](k8s/explore-kustomize/readme.md) - - [vitess](k8s/explore-vitess/readme.md) - - [k8s-list-watch](k8s/list-watch/readme.md) - - [hashicorp-nomad](k8s/hashicorp-nomad/readme.md) - - [kube-apiserver-server-chain](k8s/kube-apiserver-server-chain/readme.md) - - [Tilt Dev](k8s/explore-tilt-dev/readme.md) - - [Explore Helm](k8s/explore-helm/readme.md) - - [Helm Client in Controller](k8s/explore-helm/helm-client-in-controller/README.md) - - [multi-cluster: karmarda](k8s/explore-karmarda/readme.md) - - [Explore DRA](k8s/explore-dra/readme.md) -- K8S papers - - [Evaluating K8S Perf for GAI Inference](k8s-papers/evaluating-k8s-perf-for-gai-inference/readme.md) -- General knowledge base - - [How tree structure is stored in database](general-knowledge-base/how-tree-is-stored-in-db/readme.md) - - [Compare Redis and Memcached](general-knowledge-base/compare-redis-memcached/readme.md) - - [Distributed consensus](general-knowledge-base/distributed-consensus/readme.md) - - [Explore Oauth](general-knowledge-base/explore-oauth/readme.md) - - [Explore openLDAP](general-knowledge-base/explore-openldap/readme.md) - - [Go channels](general-knowledge-base/go-channels/readme.md) - - [Logging with cobra](general-knowledge-base/logging-with-cobra/readme.md) - - [rpc-vs-rest](general-knowledge-base/rpc-vs-rest/readme.md) - - [Distributed hash table](general-knowledge-base/distributed-hash-table/readme.md) - - [Row vs Column oriented store](general-knowledge-base/row-vs-column-oriented-database/readme.md) - - [Distributed lock](general-knowledge-base/distributed-lock/readme.md) - - [Multi-tenancy architecture](./general-knowledge-base/multi-tenancy/readme.md) -- System design - - [Distributed system architecture patterns](./system-design/distributed-system-architectural-patterns/readme.md) - - Design data intensive application notes - - [3 storage and retrieval](system-design/3-storage-and-retrieval/readme.md) - - [4 encoding and evolution](system-design/4-encoding-and-evolution/readme.md) - - [5 replication](system-design/5-replication/readme.md) - - [how to handle concurrent write](system-design/5-replication/how-to-handle-concurrent-write.md) - - [6 partitioning](system-design/6-partitioning/readme.md) - - [7 transactions](system-design/7-transactions/readme.md) - - [11 stream processing](system-design/11-stream-processing/readme.md) - - Design cases - - [Design auto complete service](system-design/topics/auto-complete-service/readme.md) - - [Design collaborative editing](system-design/topics/how-collaborative-editing-work/readme.md) - - [Design instant messaging](system-design/topics/instant-messaging-system/readme.md) - - [Design rate limiting](system-design/topics/rate-limiting/readme.md) - - [Design news feeds](system-design/topics/news-feeds/readme.md) - - [Design netflix](system-design/topics/netflix/readme.md) - - [Design distributed file system](system-design/topics/distributed-file-system/readme.md) - - [Design text based search system](system-design/topics/text-based-search/readme.md) - - [Design distributed delayed job queue](system-design/topics/distributed-delayed-job-queueing-system/readme.md) - - [Design distributed caching](system-design/topics/caching/readme.md) - - [Design real time interactions on live video](system-design/topics/realtime-interactions-on-live-video/readme.md) - - [Design message broker(RabbitMQ) or event streaming system(Kafka)](system-design/topics/message-broker-and-event-streaming/readme.md) - - [Design geolocation based service(UberEats/DoorDash/...)](system-design/topics/geolocation-based-service/readme.md) - - [Design i18n/translation service](system-design/topics/i18n-service/readme.md) - - [Design distributed web crawler](system-design/topics/distributed-web-crawler/readme.md) - - [Design stock exchange](system-design/topics/stock-exchange/readme.md) - - [Design Netflix/Youtube](system-design/topics/netflix/readme.md) - - [Design distributed counter](system-design/topics/distributed-counter/readme.md) - - [Design realtime comment](system-design/topics/realtime-comments-on-live-video/readme.md) - - [Design realtime presence](system-design/topics/realtime-presence-platform/readme.md) - - [Design monitoring system](system-design/topics/monitoring-system/prometheus) -- How Facebook xxx series - - [Cluster management: Twine](how-facebook-xxx-series/cluster-management-system/readme.md) - - [Distributed datastore for social graph: TAO](how-facebook-xxx-series/distribute-datastore-for-social-graph/readme.md) - - [Shard locality management: Akkio](how-facebook-xxx-series/managing-data-store-locality-at-scale-with-akkio/readme.md) - - [Blog: Distributed priority queuing service: FOQS](how-facebook-xxx-series/scale-a-distributed-priority-queue/readme.md) - - [Blog: Migrating Messenger storage to optimize performance](how-facebook-xxx-series/migrate-messenger-storage/readme.md) - - [Scribe: Buffered queueing system for log transporting](how-facebook-xxx-series/buffered-queueing-system-for-log-transporting/readme.md) -- How Google xxx series - - [Millwheel: Stream processing framework](how-google-xxx-series/millwheel-stream-processing-framework/readme.md) -- How Amazon xxx series - - [Dynamo](how-amazon-xxx-series/dynamo/readme.md) -- How Uber xxx series - - [Unified resource scheduler: Peloton](how-uber-xxx-series/peloton-unified-resource-scheduler/readme.md) +### AI/ML Related + +- [AI Inference Concepts](ai/ai-inference-concpets/readme.md) + - [Disaggregated Serving Concepts](ai/ai-inference-concpets/disaggregated-serving.md) +- [Build simple local RAG](ai/local-rag/readme.md) + - [Build local RAG](ai/local-rag/notebook.ipynb) +- [Try out building AI agent on local](ai/ai-agent/notebook.ipynb) + +### CKA (Certified Kubernetes Administrator) Related + +- [Create the K8S cluster on vSphere the hard way](cka/create-k8s-on-vsphere-hard-way.md) +- [Create user accounts](cka/create-user-accounts/create-user-accounts.md) +- [Init containers](cka/manage-pod/init-containers.md) +- [Manage custome resources](cka/manage-custome-resources/manage-custome-resources.md) +- [Manage pod networking](cka/manage-pod-networking/manage-pod-networking.md) +- [Manage scheduling](cka/manage-scheduling/manage-scheduling.md) +- [Mange cluster nodes](cka/manage-cluster-nodes/manage-cluster-nodes.md) +- [Monitoring K8S resources](cka/monitoring/monitor-k8s-resources.md) +- [Pod volumes](cka/manage-volumes/pod-volumes.md) +- [Sample exam](cka/sample-exam/sample-exam.md) + +### Kubernetes CNCF Projects Related + +- [Deep dive admission controllers](k8s/explore-admission-controllers/readme.md) +- [Explore Antrea](k8s/explore-antrea/readme.md) +- [Explore cert manager](k8s/explore-cert-manager/readme.md) +- [Explore cluster API for docker infrastructure](k8s/explore-capd/readme.md) +- [Explore custom controller leader election](k8s/explore-controller-leader-election/readme.md) +- [Explore DRA](k8s/explore-dra/readme.md) +- [Explore ETCD](k8s/explore-etcd/readme.md) +- [Explore Helm](k8s/explore-helm/readme.md) + - [helm-client-in-controller](k8s/explore-helm/helm-client-in-controller/readme.md) +- [Explore k14s](k8s/explore-k14s/readme.md) +- [Explore kapp controller](k8s/explore-kapp-controller/readme.md) +- [Explore Karmarda (WIP)](k8s/explore-karmarda/readme.md) +- [Explore Kata Container](k8s/explore-kata-container/readme.md) +- [Explore kubevirt](k8s/explore-kubevirt/readme.md) +- [Explore KUDO](k8s/explore-kudo/readme.md) +- [Explore Kustomize](k8s/explore-kustomize/readme.md) +- [Explore pinniped](k8s/explore-pinniped/readme.md) + - [Authentication in K8S](k8s/explore-pinniped/auth-in-k8s.md) +- [Explore secrets store csi](k8s/explore-secrets-store-csi/explore-secrets-store-csi.md) +- [Explore Tilt Dev's live updates](k8s/explore-tilt-dev/readme.md) +- [Explore Vitess](k8s/explore-vitess/readme.md) +- [Expore Rancher](k8s/explore-rancher/readme.md) +- [Hashicorp Nomad](k8s/hashicorp-nomad/readme.md) +- [How pod is created via Deployment with the network configured](k8s/how-pod-created-with-network-configured/readme.md) +- [kube-apiserver-server-chain](k8s/kube-apiserver-server-chain/readme.md) +- [List-Watch](k8s/list-watch/readme.md) +- [Pod cold start performance](k8s/pod-cold-start-performance/readme.md) +- [StatefulSets](k8s/explore-statefulset-on-vsphere/explore-statefulset-on-vsphere.md) + +### Kubernetes Related Papers + +- [[Paper] Evaluating Kubernetes Performance for GAI Inference](k8s-papers/evaluating-k8s-perf-for-gai-inference/readme.md) + +### General Knowledge Base + +- [Compare Redis and Memcached](general-knowledge-base/compare-redis-memcached/readme.md) +- [Conflict-free Replicated Date Types](general-knowledge-base/conflict-free-replicated-data-types/readme.md) +- [Distributed consensus](general-knowledge-base/distributed-consensus/readme.md) + - [Deep dive into config change in distributed system](general-knowledge-base/distributed-consensus/deep-dive-config-change.md) + - [Raft distributed consensus](general-knowledge-base/distributed-consensus/raft-distributed-consensus.md) +- [Distributed hash table](general-knowledge-base/distributed-hash-table/readme.md) +- [Distributed lock](general-knowledge-base/distributed-lock/readme.md) +- [Explore Confidential VMs](general-knowledge-base/confidential-vm/readme.md) +- [Explore Google SaaS Runtime](general-knowledge-base/google-saas-runtime/readme.md) +- [Explore OAuth](general-knowledge-base/explore-oauth/readme.md) +- [Explore OpenLDAP](general-knowledge-base/explore-openldap/readme.md) +- [Go channels](general-knowledge-base/go-channels/readme.md) +- [How tree is stored in database](general-knowledge-base/how-tree-is-stored-in-db/readme.md) +- [Idempotency](general-knowledge-base/idempotency/readme.md) +- [Logging in Cobra](general-knowledge-base/logging-with-cobra/readme.md) +- [Multi-tenancy](general-knowledge-base/multi-tenancy/readme.md) +- [Optimistic vs Pessimistic concurrency control](general-knowledge-base/optimistic-pessimistic-concurrency-control/readme.md) +- [Row vs Column oriented databases](general-knowledge-base/row-vs-column-oriented-database/readme.md) +- [RPC vs REST](general-knowledge-base/rpc-vs-rest/readme.md) +- [Something about gossip-protocol](general-knowledge-base/something-about-gossip-protocols/readme.md) +- [UUID](general-knowledge-base/uuid/readme.md) + +### System Design + +- Architecture & Patterns + - [Distributed systems architectural patterns](system-design/distributed-system-architectural-patterns/readme.md) +- Data-Intensive Applications (Book Notes) + - [Storage and retrieval](system-design/3-storage-and-retrieval/readme.md) + - [Column oriented storage](system-design/3-storage-and-retrieval/column-oriented-storage.md) + - [Compare between B-Tree and LSM tree](system-design/3-storage-and-retrieval/compare-b-tree-vs-lsm-tree.md) + - [Log structured storage](system-design/3-storage-and-retrieval/log-structured-storage.md) + - [OLTP VS OLAP](system-design/3-storage-and-retrieval/oltp-vs-olap.md) + - [Other indexing structures](system-design/3-storage-and-retrieval/other-indexing-structures.md) + - [Page oriented storage](system-design/3-storage-and-retrieval/page-oriented-storage.md) + - [Encoding and evolutiion](system-design/4-encoding-and-evolution/readme.md) + - [Replication](system-design/5-replication/readme.md) + - [How to handle concurrent write](system-design/5-replication/how-to-handle-concurrent-write.md) + - [Anti-Entropy](system-design/5-replication/manage-anti-entropy-with-merkle-tree.md) + - [Partitioning](system-design/6-partitioning/readme.md) + - [Trasactions](system-design/7-transactions/readme.md) + - [Consistency and consensus](system-design/9-consistency-and-consensus/readme.md) + - [Stream processing](system-design/11-stream-processing/readme.md) +- System Design Case Studies + - [Auto complete service](system-design/topics/auto-complete-service/readme.md) + - [Caching](system-design/topics/caching/readme.md) + - [Collaborative editing](system-design/topics/how-collaborative-editing-work/readme.md) + - [Design a distributed delayed job queueing system](system-design/topics/distributed-delayed-job-queueing-system/readme.md) + - [Design a key-value store](system-design/topics/distributed-key-value-store/readme.md) + - [Design distributed message broker(RabbitMQ) and message streaming platform(Kafka)](system-design/topics/message-broker-and-event-streaming/readme.md) + - [Design i18n service](system-design/topics/i18n-service/readme.md) + - [Design instant messaging system](system-design/topics/instant-messaging-system/readme.md) + - [Design Netflix or Youtube](system-design/topics/netflix/readme.md) + - [Design news feeds system](system-design/topics/news-feeds/readme.md) + - [Design payment system](system-design/topics/payment-system/readme.md) + - [Design text based search service](system-design/topics/text-based-search/readme.md) + - [Distributed counter](system-design/topics/distributed-counter/readme.md) + - [CRDT Distributed Counter](system-design/topics/distributed-counter/crdt-distributed-counter.md) + - [Distributed file system](system-design/topics/distributed-file-system/readme.md) + - [Distributed log aggregation](system-design/topics/distributed-log-aggregation/readme.md) + - [Distributed monitoring system](system-design/topics/monitoring-system/readme.md) + - [Mimir](system-design/topics/monitoring-system/mimir.md) + - [Prometheus](system-design/topics/monitoring-system/prometheus.md) + - [TSDB](system-design/topics/monitoring-system/tsdb.md) + - [Distributed Unique ID](system-design/topics/distributed-unique-id/readme.md) + - [Distributed web crawler](system-design/topics/distributed-web-crawler/readme.md) + - [Geolocation based service](system-design/topics/geolocation-based-service/readme.md) + - [Rate limiting](system-design/topics/rate-limiting/readme.md) + - [Real time interactions on live video](system-design/topics/realtime-interactions-on-live-video/readme.md) + - [Realtime comments on live video](system-design/topics/realtime-comments-on-live-video/readme.md) + - [Long Polling vs SSE vs WebSocket](system-design/topics/realtime-comments-on-live-video/long-polling-vs-sse-vs-websocket.md) + - [Realtime gaming leaderboard](system-design/topics/realtime-gaming-leaderboard/readme.md) + - [Realtime presence platform](system-design/topics/realtime-presence-platform/readme.md) + - [Stock Exchange](system-design/topics/stock-exchange/readme.md) + - [What questions to ask at the beginning of a design](system-design/topics/what-to-ask-at-beginning-of-a-design.md) + +### How Facebook Builds Systems + +- [FOQS: Scaling a distributed priority queue](how-facebook-xxx-series/scale-a-distributed-priority-queue/readme.md) +- [Gorilla: A Fast, Scalable, In-memory Time Series Database](how-facebook-xxx-series/gorilla-in-memory-tsdb/readme.md) +- [Manage datastore locality at scale with Akkio](how-facebook-xxx-series/managing-data-store-locality-at-scale-with-akkio/readme.md) +- [Migrating Messenger storage to optimize performance](how-facebook-xxx-series/migrate-messenger-storage/readme.md) +- [Scribe: Transporting petabytes per hour via a distributed, buffered queueing system](how-facebook-xxx-series/buffered-queueing-system-for-log-transporting/readme.md) +- [TAO: Facebook’s Distributed Data Store for the Social Graph](how-facebook-xxx-series/distribute-datastore-for-social-graph/readme.md) +- [Twine: A Unified Cluster Management System for Shared Infrastructure](how-facebook-xxx-series/cluster-management-system/readme.md) + +### How Google Builds Systems + +- [MillWheel: Fault-Tolerant Stream Processing at Internet Scale](how-google-xxx-series/millwheel-stream-processing-framework/readme.md) +- [Monarch: Google's Planet-Scale In-Memory Time Series Database](how-google-xxx-series/monarch-planet-scale-in-memory-time-series-database/readme.md) + +### How Amazon Builds Systems + +- [Dynamo](how-amazon-xxx-series/dynamo/readme.md) + +### How Uber Builds Systems + +- [Peloton: Uber’s Unified Resource Scheduler for Diverse Cluster Workloads](how-uber-xxx-series/peloton-unified-resource-scheduler/readme.md) + +### How Alibaba Builds Systems + +- [Virtual Control Plane implementation: A Multi-Tenant Framework for Cloud Container Services](how-alibaba-xxx-series/virtual-control-plane/readme.md) + diff --git a/ai/ai-agent/notebook.ipynb b/ai/ai-agent/notebook.ipynb index 507d287..80acceb 100644 --- a/ai/ai-agent/notebook.ipynb +++ b/ai/ai-agent/notebook.ipynb @@ -4,7 +4,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Goals\n", + "# Try out building AI agent on local" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Goals\n", "I would like to have an chat model that can parse the user input and autonomously manage sequential tool invocations to complete the task. " ] }, diff --git a/generate_toc.py b/generate_toc.py new file mode 100755 index 0000000..0860fa4 --- /dev/null +++ b/generate_toc.py @@ -0,0 +1,294 @@ +#!/usr/bin/env python3 +import os +import json +import re +import sys + +workspace = os.path.abspath(os.path.dirname(__file__)) + +CATEGORIES = { + "ai": "AI/ML Related", + "cka": "CKA (Certified Kubernetes Administrator) Related", + "k8s": "Kubernetes CNCF Projects Related", + "k8s-papers": "Kubernetes Related Papers", + "general-knowledge-base": "General Knowledge Base", + "system-design": "System Design", + "how-facebook-xxx-series": "How Facebook Builds Systems", + "how-google-xxx-series": "How Google Builds Systems", + "how-amazon-xxx-series": "How Amazon Builds Systems", + "how-uber-xxx-series": "How Uber Builds Systems", + "how-alibaba-xxx-series": "How Alibaba Builds Systems", +} + +def parse_title(file_path): + ext = os.path.splitext(file_path)[1].lower() + if ext == '.md': + try: + with open(file_path, 'r', encoding='utf-8') as f: + for line in f: + line = line.strip() + if line.startswith('# '): + title = line[2:].strip() + if title.startswith('**') and title.endswith('**'): + title = title[2:-2].strip() + return title + except Exception: + pass + elif ext == '.ipynb': + try: + with open(file_path, 'r', encoding='utf-8') as f: + data = json.load(f) + for cell in data.get('cells', []): + if cell.get('cell_type') == 'markdown': + for line in cell.get('source', []): + line = line.strip() + if line.startswith('# '): + title = line[2:].strip() + if title.startswith('**') and title.endswith('**'): + title = title[2:-2].strip() + return title + except Exception: + pass + return None + +def humanize(name): + parts = name.replace('-', ' ').replace('_', ' ').split() + acronyms = { + 'ai': 'AI', 'ml': 'ML', 'rag': 'RAG', 'k8s': 'Kubernetes', 'cka': 'CKA', + 'oauth': 'OAuth', 'db': 'Database', 'vm': 'VM', 'saas': 'SaaS', 'csi': 'CSI', + 'gai': 'GAI', 'dra': 'DRA', 'tsdb': 'TSDB', 'uuid': 'UUID', 'rpc': 'RPC', + 'rest': 'REST', 'crdt': 'CRDT', 'sse': 'SSE', 'i18n': 'I18n' + } + capitalized = [] + for p in parts: + lower_p = p.lower() + if lower_p in acronyms: + capitalized.append(acronyms[lower_p]) + else: + capitalized.append(p.capitalize()) + return ' '.join(capitalized) + +def get_note_info(rel_dir): + full_dir = os.path.join(workspace, rel_dir) + if not os.path.isdir(full_dir): + return None + + files = [] + for entry in os.scandir(full_dir): + if entry.is_file() and entry.name.lower().endswith(('.md', '.ipynb')): + files.append(entry.name) + + if not files: + return None + + primary = None + for name in ['readme.md', 'README.md', 'notebook.ipynb']: + if name in files: + primary = name + break + if not primary: + md_files = [f for f in files if f.lower().endswith('.md')] + if md_files: + primary = sorted(md_files)[0] + else: + primary = sorted(files)[0] + + secondary = sorted([f for f in files if f != primary]) + primary_path = os.path.join(full_dir, primary) + title = parse_title(primary_path) + if not title: + title = humanize(os.path.basename(rel_dir)) + + return { + "dir": rel_dir, + "primary_file": os.path.join(rel_dir, primary), + "title": title, + "secondary_files": [os.path.join(rel_dir, f) for f in secondary] + } + +def process_category(cat_name): + cat_dir = os.path.join(workspace, cat_name) + if not os.path.exists(cat_dir): + return [] + + items = [] + + if cat_name == "system-design": + chapters = [] + topics = [] + general = [] + + for entry in os.scandir(cat_dir): + if entry.is_dir(): + if entry.name.startswith(('.', '_')): + continue + if entry.name == "topics": + topics_dir = os.path.join(cat_dir, "topics") + for sub in os.scandir(topics_dir): + if sub.is_dir() and not sub.name.startswith('.'): + note = get_note_info(os.path.join("system-design", "topics", sub.name)) + if note: + topics.append(note) + elif sub.is_file() and sub.name.lower().endswith(('.md', '.ipynb')): + sub_title = parse_title(sub.path) or humanize(os.path.splitext(sub.name)[0]) + topics.append({ + "dir": "system-design/topics", + "primary_file": os.path.join("system-design", "topics", sub.name), + "title": sub_title, + "secondary_files": [] + }) + elif re.match(r'^\d+', entry.name): + note = get_note_info(os.path.join("system-design", entry.name)) + if note: + chapters.append(note) + else: + note = get_note_info(os.path.join("system-design", entry.name)) + if note: + general.append(note) + + chapters.sort(key=lambda x: int(re.match(r'^\d+', os.path.basename(x['dir'])).group())) + topics.sort(key=lambda x: x['title'].lower()) + general.sort(key=lambda x: x['title'].lower()) + + return { + "type": "system-design", + "chapters": chapters, + "topics": topics, + "general": general + } + + if cat_name == "cka": + direct_files = [] + subdirs = [] + for entry in os.scandir(cat_dir): + if entry.is_file() and entry.name.lower().endswith(('.md', '.ipynb')): + title = parse_title(entry.path) or humanize(os.path.splitext(entry.name)[0]) + direct_files.append({ + "primary_file": os.path.join("cka", entry.name), + "title": title, + "secondary_files": [] + }) + elif entry.is_dir() and not entry.name.startswith('.'): + note = get_note_info(os.path.join("cka", entry.name)) + if note: + subdirs.append(note) + + direct_files.sort(key=lambda x: x['title'].lower()) + subdirs.sort(key=lambda x: x['title'].lower()) + return { + "type": "cka", + "direct_files": direct_files, + "subdirs": subdirs + } + + for entry in os.scandir(cat_dir): + if entry.is_dir(): + if entry.name.startswith(('.', '_')): + continue + note = get_note_info(os.path.join(cat_name, entry.name)) + if note: + nested_dir_path = os.path.join(workspace, cat_name, entry.name) + nested_notes = [] + for sub_entry in os.scandir(nested_dir_path): + if sub_entry.is_dir() and not sub_entry.name.startswith(('.', '_', 'resources', 'resource')): + nested_note = get_note_info(os.path.join(cat_name, entry.name, sub_entry.name)) + if nested_note: + nested_notes.append(nested_note) + if nested_notes: + nested_notes.sort(key=lambda x: x['title'].lower()) + note['nested_notes'] = nested_notes + items.append(note) + elif entry.is_file() and entry.name.lower().endswith(('.md', '.ipynb')) and entry.name.lower() not in ['readme.md', 'license']: + title = parse_title(entry.path) or humanize(os.path.splitext(entry.name)[0]) + items.append({ + "dir": cat_name, + "primary_file": os.path.join(cat_name, entry.name), + "title": title, + "secondary_files": [] + }) + + items.sort(key=lambda x: x['title'].lower()) + return { + "type": "regular", + "items": items + } + +def generate_toc_markdown(): + lines = [] + for cat_key, cat_title in CATEGORIES.items(): + lines.append(f"### {cat_title}\n\n") + data = process_category(cat_key) + if not data: + lines.append("*(No notes found)*\n\n") + continue + + if data["type"] == "system-design": + if data["general"]: + lines.append("- Architecture & Patterns\n") + for item in data["general"]: + lines.append(f" - [{item['title']}]({item['primary_file']})\n") + if data["chapters"]: + lines.append("- Data-Intensive Applications (Book Notes)\n") + for item in data["chapters"]: + lines.append(f" - [{item['title']}]({item['primary_file']})\n") + for sec in item['secondary_files']: + sec_title = parse_title(os.path.join(workspace, sec)) or humanize(os.path.splitext(os.path.basename(sec))[0]) + lines.append(f" - [{sec_title}]({sec})\n") + if data["topics"]: + lines.append("- System Design Case Studies\n") + for item in data["topics"]: + lines.append(f" - [{item['title']}]({item['primary_file']})\n") + for sec in item['secondary_files']: + sec_title = parse_title(os.path.join(workspace, sec)) or humanize(os.path.splitext(os.path.basename(sec))[0]) + lines.append(f" - [{sec_title}]({sec})\n") + + elif data["type"] == "cka": + for item in data["direct_files"]: + lines.append(f"- [{item['title']}]({item['primary_file']})\n") + for item in data["subdirs"]: + lines.append(f"- [{item['title']}]({item['primary_file']})\n") + for sec in item['secondary_files']: + sec_title = parse_title(os.path.join(workspace, sec)) or humanize(os.path.splitext(os.path.basename(sec))[0]) + lines.append(f" - [{sec_title}]({sec})\n") + + else: + for item in data["items"]: + lines.append(f"- [{item['title']}]({item['primary_file']})\n") + for sec in item.get('secondary_files', []): + sec_title = parse_title(os.path.join(workspace, sec)) or humanize(os.path.splitext(os.path.basename(sec))[0]) + lines.append(f" - [{sec_title}]({sec})\n") + for nest in item.get('nested_notes', []): + lines.append(f" - [{nest['title']}]({nest['primary_file']})\n") + for sec in nest.get('secondary_files', []): + sec_title = parse_title(os.path.join(workspace, sec)) or humanize(os.path.splitext(os.path.basename(sec))[0]) + lines.append(f" - [{sec_title}]({sec})\n") + lines.append("\n") + return "".join(lines) + +def main(): + dry_run = "--dry-run" in sys.argv + + readme_path = os.path.join(workspace, "README.md") + if not os.path.exists(readme_path): + header = "# Tech Notes\n\n## Table of contents\n\n" + else: + with open(readme_path, 'r', encoding='utf-8') as f: + current_content = f.read() + toc_marker = "## Table of contents" + if toc_marker in current_content: + header = current_content.split(toc_marker)[0] + toc_marker + "\n\n" + else: + header = current_content + "\n\n## Table of contents\n\n" + + toc_content = generate_toc_markdown() + new_content = header + toc_content + + if dry_run: + print(new_content) + else: + with open(readme_path, 'w', encoding='utf-8') as f: + f.write(new_content) + print("Successfully updated README.md Table of Contents!") + +if __name__ == "__main__": + main() diff --git a/k8s/explore-admission-controllers/explore-admission-controllers.md b/k8s/explore-admission-controllers/readme.md similarity index 100% rename from k8s/explore-admission-controllers/explore-admission-controllers.md rename to k8s/explore-admission-controllers/readme.md diff --git a/k8s/explore-antrea/explore-antrea.md b/k8s/explore-antrea/readme.md similarity index 100% rename from k8s/explore-antrea/explore-antrea.md rename to k8s/explore-antrea/readme.md diff --git a/k8s/explore-capd/explore-capd.md b/k8s/explore-capd/readme.md similarity index 100% rename from k8s/explore-capd/explore-capd.md rename to k8s/explore-capd/readme.md diff --git a/k8s/explore-cert-manager/explore-cert-manager.md b/k8s/explore-cert-manager/readme.md similarity index 100% rename from k8s/explore-cert-manager/explore-cert-manager.md rename to k8s/explore-cert-manager/readme.md diff --git a/k8s/explore-helm/helm-client-in-controller/README.md b/k8s/explore-helm/helm-client-in-controller/readme.md similarity index 100% rename from k8s/explore-helm/helm-client-in-controller/README.md rename to k8s/explore-helm/helm-client-in-controller/readme.md diff --git a/k8s/explore-kubevirt/explore-kubevirt.md b/k8s/explore-kubevirt/readme.md similarity index 100% rename from k8s/explore-kubevirt/explore-kubevirt.md rename to k8s/explore-kubevirt/readme.md diff --git a/k8s/explore-kudo/explore-kudo.md b/k8s/explore-kudo/readme.md similarity index 100% rename from k8s/explore-kudo/explore-kudo.md rename to k8s/explore-kudo/readme.md diff --git a/k8s/explore-rancher/explore-rancher.md b/k8s/explore-rancher/readme.md similarity index 100% rename from k8s/explore-rancher/explore-rancher.md rename to k8s/explore-rancher/readme.md