Command line application to download youtube videos.
positional arguments: url The YouTube /watch or /playlist url
optional arguments: -h, --help show this help message and exit --version show program's version number and exit --itag ITAG The itag for the desired stream -r RESOLUTION, --resolution RESOLUTION The resolution forthe desired stream -l, --list The list option causes pytube cli to return a list of streams available to download -v, --verbose Verbosity level, use up to 4 to increase logging -vvvv --logfile LOGFILE logging debug and error messages into a log file --build-playback-report Save the html and js to disk -c CAPTION_CODE, --caption-code CAPTION_CODE Download srt captions for given language code. Prints available language codes if no argument given -lc, --list-captions List available caption codes for a video -t TARGET, --target TARGET The output directoryforthe downloaded stream. Default is current working directory -a [AUDIO], --audio [AUDIO] Download the audio fora given URLatthe highest bitrate availableDefaults to mp4 formatifnone is specified -f [FFMPEG], --ffmpeg [FFMPEG] Downloads the audio and video stream for resolution providedIf no resolution is provided, downloads the best resolutionRuns thecommandlineprogramffmpegto combinethe audio and video
2 00:00:13,400 --> 00:00:16,200 That is so awkward to watch. ...
对于播放列表的处理，比如新建 Playlist 对象，然后取出每一个视频的第一个视频流并下载：
1 2 3 4
>>> from pytube import Playlist >>> p = Playlist('https://www.youtube.com/playlist?list=PLS1QulWo1RIaJECMeUT4LFwJ-ghgoSH6n') >>> for video in p.videos: >>> video.streams.first().download()
另外还有一些异常处理机制：
1 2 3 4 5 6 7 8 9 10 11
>>> from pytube import Playlist, YouTube >>> playlist_url = 'https://youtube.com/playlist?list=special_playlist_id' >>> p = Playlist(playlist_url) >>> for url in p.video_urls: ... try: ... yt = YouTube(url) ... except VideoUnavailable: ... print(f'Video {url} is unavaialable, skipping.') ... else: ... print(f'Downloading video: {url}') ... yt.streams.first().download()
Github1s is based on VS Code 1.52.1 now. VS Code can be built for a browser version officially. I also used the code and got inspired by Code Server.
Thanks to the very powerful and flexible extensibility of VS Code, we can easily implement a VS Code extension that provides the custom File IO ability using FileSystemProvider API. There is an official demo named vscode-web-playground which shows how it is used.
On the other hand, GitHub provides the powerful REST API that can be used for a variety of tasks which includes reading directories and files for sure.
According to the above, obviously, the core concept of GitHub1s is to implement a VS Code Extension (includes FileSystemProvider) using GitHub REST API.
We may switch to the GitHub GraphQL API for more friendly user experience in the future, thanks to @xcv58 and @kanhegaonkarsaurabh. See details at Issue 12.
GitHub1s is a purely static web app (because it really doesn’t need a backend service, does it?). So we just deploy it on GitHub Pages now (the gh-pages branch of this repository), and it is free. The service of GitHub1s could be reliable (GitHub is very reliable) because nobody needs to pay the web hosting bills.
他基于 VS Code 提供的 FileSystemProvider API 对接了 GitHub 的 REST API 实现了这些功能。其中前者是 VS Code 提供的，可以提供文件读写操作，当然读写在线文件也是没问题的了；而后者是 GitHub 提供的，通过 REST API 可以获取 Repo 的文件夹或者某个文件。
For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the user making requests.
对于未授权的请求，API 的请求频率是有限制的，每个 IP 每个小时访问限制是 60 次，所以用着用着就容易超限制了，可能就打不开文件了。
CORS Anywhere is a NodeJS proxy which adds CORS headers to the proxied request.
The url to proxy is literally taken from the path, validated and proxied. The protocol part of the proxied URI is optional, and defaults to “http”. If port 443 is specified, the protocol defaults to “https”.
This package does not put any restrictions on the http methods or headers, except for cookies. Requesting user credentials is disallowed. The app can be configured to require a header for proxying a request, for example to avoid a direct visit from the browser.
真是一个开源框架，和我猜的一样，就是一个解决跨域问题而生的反向代理。
然后我就在它的 README 中看到了这个：
好家伙，这不就是我刚才用到的链接吗？
那肯定是这个玩意出了什么毛病。
咋看呢？这个果断就是找 Issue 了：
一看，太明显了：
PSA: Public demo server (cors-anywhere.herokuapp.com) will be very limited by January 2021, 31st
意思就是从今年 1.31 开始这个网站的访问会受限，点进去看看：
The demo server of CORS Anywhere (cors-anywhere.herokuapp.com) is meant to be a demo of this project. But abuse has become so common that the platform where the demo is hosted (Heroku) has asked me to shut down the server, despite efforts to counter the abuse (rate limits in #45 and #164, and blocking other forms of requests). Downtime becomes increasingly frequent (e.g. recently #300, #299, #295, #294, #287) due to abuse and its popularity.
To counter this, I will make the following changes:
The rate limit will decrease from 200 (#164) per hour to 50 per hour.
By January 31st, 2021, cors-anywhere.herokuapp.com will stop serving as an open proxy.
From February 1st. 2021, cors-anywhere.herokuapp.com will only serve requests after the visitor has completed a challenge: The user (developer) must visit a page at cors-anywhere.herokuapp.com to temporarily unlock the demo for their browser. This allows developers to try out the functionality, to help with deciding on self-hosting or looking for alternatives.
-name:Autogreen run:| gitconfig--localuser.email"justjavac@gmail.com" gitconfig--localuser.name"迷渡" gitremoteset-urloriginhttps://${{github.actor}}:${{secrets.GITHUB_TOKEN}}@github.com/${{github.repository}} gitpull--rebase gitcommit--allow-empty-m"a commit a day keeps your girlfriend away" gitpush
const item = new clipboard.ClipboardItem({ "text/html": new Blob( ["<i>Markup</i> <b>text</b>. Paste me into a rich text editor."], { type: "text/html" } ), "text/plain": new Blob( ["Fallback markup text. Paste me into a rich text editor."], { type: "text/plain" } ), }); await clipboard.write([item]); }
def get_client_ip(request): x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR') if x_forwarded_for: ip = x_forwarded_for.split(',')[0] else: ip = request.META.get('REMOTE_ADDR') return ip
functionprintLabel(labeledObj: { label: number }) { console.log(labeledObj.label); }
则会出现如下报错：
1
Argument of type '{ size: number; label: string; }' is not assignable to parameter of type '{ label: number; }'. Types of property 'label' are incompatible. Type 'string' is not assignable to type 'number'.
Argument of type '{ size: number; count: number; label: string; }' is not assignable to parameter of type 'LabeledValue'. Property 'message' is missing in type '{ size: number; count: number; label: string; }' but required in type 'LabeledValue'.
let a: number[] = [1, 2, 3, 4]; let ro: ReadonlyArray<number> = a;
ro[0] = 12; // error! Index signature intype'readonly number[]' only permits reading. ro.push(5); // error! Property 'push' does not exist on type'readonly number[]'. ro.length = 100; // error! Cannot assign to 'length' because it is a read-only property. a = ro; // error! The type'readonly number[]' is 'readonly' and cannot be assigned to the mutable type'number[]'.
interface SelectableControl extends Control { select(): void; }
class Button extends Control implements SelectableControl { select() {} }
class TextBox extends Control { select() {} }
class ImageControl implements SelectableControl { // Error, Class 'ImageControl' incorrectly implements interface 'SelectableControl'. // Types have separate declarations of a private property 'state'. private state: any; select() {} }
上面我们可以知道，当创建一个扩展带有私有或受保护成员的 Class 的 Interface 时，该 Interface 只能由该 Class 或其子 Class 实现。在这里 ImageControl 由于没有继承 Control，但同时 Control 还包含了私有成员变量，所以 ImageControl 并不能继承得到 state 这个私有成员变量，所以会报错。
TypeScript extends JavaScript by adding types. By understanding JavaScript, TypeScript saves you time catching errors and providing fixes before you run code. Any browser, any OS, anywhere JavaScript runs. Entirely Open Source.
let color: string = "blue"; color = 'red'; let fullName: string = `Bob Bobbington`; let age: number = 37; let sentence: string = `Hello, my name is ${fullName}. I'll be ${age + 1} years old next month.`; console.log(sentence, typeof sentence);
运行结果如下：
1
[LOG]: "Hello, my name is Bob Bobbington. I'll be 38 years old next month.", "string"
let list: number[] = [1, 2, 3]; let list2: Array<number> = [1, 2, 3]; let list3: Array<any> = [1, "2", 3]; console.log(list, typeof list); console.log(list2, typeof list2); console.log(list3, typeof list3);
其中 list 就是使用了 type[] 这样的声明方式，声明为 number[]，那么数组里面的每个元素都必须要是 number 类型，list2 则是使用了泛型类型的声明，和 list 的效果是一样的，另外 list3 使用了 any 泛型类型，所以其值可以不仅仅为 number，因此这里 list3 的第二个元素设置为了字符串 2。
declareconst maybe: unknown; // 'maybe' could be a string, object, boolean, undefined, or other types const aNumber: number = maybe; // Type 'unknown' is not assignable to type 'number'.
if (maybe === true) { // TypeScript knows that maybe is a boolean now const aBoolean: boolean = maybe; // So, it cannot be a string const aString: string = maybe; // Type 'boolean' is not assignable to type 'string'. }
if (typeof maybe === "string") { // TypeScript knows that maybe is a string const aString: string = maybe; // So, it cannot be a boolean const aBoolean: boolean = maybe; // Type 'string' is not assignable to type 'boolean'. }
// OK create({ prop: 0 }); create(null); create(42); // Error, Argument of type '42' is not assignable to parameter of type 'object | null'. create("string"); // Error, Argument of type '"string"' is not assignable to parameter of type 'object | null'. create(false); // Error, Argument of type 'false' is not assignable to parameter of type 'object | null'. create(undefined); // Error, Argument of type 'undefined' is not assignable to parameter of type 'object | null'.
let someValue: unknown = "this is a string"; let strLength: number = (someValue asstring).length; let someValue2: unknown = "this is a string"; let strLength2: number = (<string>someValue2).length;
这里有两种使用方式，一种是 as，一种是尖括号声明。
注意
另外值得注意到是，以上的一些类型声明，使用大写形式 Number, String, Boolean, Symbol and Object 也是可以的，不过不推荐这么做，推荐还是用小写的形式。
ubuntu@VM-0-2-ubuntu:/var/lib/docker/containers$sudofdisk-l Disk /dev/vda:100GiB,107374182400bytes,209715200sectors Units:sectorsof1*512=512bytes Sectorsize(logical/physical):512bytes/512bytes I/Osize(minimum/optimal):512bytes/512bytes Disklabel type:dos Disk identifier:0x3fa1d255
Disk /dev/vdb:200GiB,214748364800bytes,419430400sectors Units:sectorsof1*512=512bytes Sectorsize(logical/physical):512bytes/512bytes I/Osize(minimum/optimal):512bytes/512bytes
这是因为之前的数据被格式化为了 ext4，但是新扩容的部分并没有。
问了腾讯云的工程师，这时候必须要重新格式化才能用上新的容量，没有好的解决方法。
最后的解决方案：
将当前节点所有 Pod 驱逐到其他的节点。
当前节点退出集群然后配置磁盘，重新操作 mount 和格式化。（但是这里劝别折腾了，终究还是要格式化的。
他的 Title 叫 DISTINUISHED ENGINEER，这个 Title 可不是一般的牛逼，在微软，有这样的 Title 的人可是屈指可数的，DISTINUISHED 意思就是杰出的，非常牛逼的意思，能有这种 Title 的是为业界或公司做出过特别特别大贡献的，特别有影响力的，可能普通人在微软呆个二十多年都不一定能到这个地位。他的职级和 Report Line 就不说了，他距离微软 CEO 纳德拉只差 3 级，稍微形象点说就是对标阿里 P11 或者 P12 的位置吧。
他的部门就直接写了 Python and Tools for AI，和 AI 相关，同时 Guido 又这么热爱开源，微软也在一直拥抱开源，期待 Python 之父将来不久之后又出新的杰作吧。
import cchardet from retrying import retry from powerspider import logger from powerspider.tools.Ua import ua from requests import request, RequestException
@retry(stop_max_attempt_number=3, retry_on_result=lambda x: x is None, wait_fixed=2000) defdownloader(url, method=None, header=None, timeout=None, binary=False, **kwargs): logger.info(f'Scraping {url}') _header = {'User-Agent': ua()} _maxTimeout = timeout if timeout else5 _headers = header if header else _header _method = "GET"ifnot method else method try: response = request(method=_method, url=url, headers=_headers, **kwargs) encoding = cchardet.detect(response.content)['encoding'] if response.status_code == 200: return response.content if binary else response.content.decode(encoding) elif200 < response.status_code < 400: logger.info(f"Redirect_URL: {response.url}") logger.error('Get invalid status code %s while scraping %s', response.status_code, url) except RequestException as e: logger.error(f'Error occurred while scraping {url}, Msg: {e}', exc_info=True)
if __name__ == '__main__': print(downloader("https://www.baidu.com/", "GET"))
Basic Commands (Beginner): create # Create a resource from a file or from stdin. expose # 使用 replication controller, service, deployment 或者 pod 并暴露它作为一个 新的 Kubernetes Service run # 在集群中运行一个指定的镜像 set# 为 objects 设置一个指定的特征
Basic Commands (Intermediate): explain # 查看资源的文档 get # 显示一个或更多 resources edit # 在服务器上编辑一个资源 delete # Delete resources by filenames, stdin, resources and names, or by resources and label selector
Deploy Commands: rollout # Manage the rollout of a resource scale # 为 Deployment, ReplicaSet, Replication Controller 或者 Job 设置一个新的副本数量 autoscale # 自动调整一个 Deployment, ReplicaSet, 或者 ReplicationController 的副本数量
Troubleshooting and Debugging Commands: describe # 显示一个指定 resource 或者 group 的 resources 详情 logs # 输出容器在 pod 中的日志 attach # Attach 到一个运行中的 container exec# 在一个 container 中执行一个命令 port-forward # Forward one or more local ports to a pod proxy # 运行一个 proxy 到 Kubernetes API server cp # 复制 files 和 directories 到 containers 和从容器中复制 files 和 directories. auth # Inspect authorization
Advanced Commands: diff # Diff live version against would-be applied version apply # 通过文件名或标准输入流(stdin)对资源进行配置 patch # 使用 strategic merge patch 更新一个资源的 field(s) replace # 通过 filename 或者 stdin替换一个资源 wait# Experimental: Wait for a specific condition on one or many resources. convert # 在不同的 API versions 转换配置文件 kustomize # Build a kustomization target from a directory or a remote url.
Settings Commands: label # 更新在这个资源上的 labels annotate # 更新一个资源的注解 completion # Output shell completion code for the specified shell (bash or zsh)
Other Commands: api-resources # Print the supported API resources on the server api-versions # Print the supported API versions on the server, in the form of "group/version" config # 修改 kubeconfig 文件 plugin # Provides utilities for interacting with plugins. version # 输出 client 和 server 的版本信息
Examples: # Create a pod using the data in pod.json. kubectl create -f ./pod.json # Create a pod based on the JSON passed into stdin. cat pod.json | kubectl create -f - # Edit the data in docker-registry.yaml in JSON then create the resource using the edited data. kubectl create -f docker-registry.yaml --edit -o json
Available Commands: clusterrole # Create a ClusterRole. clusterrolebinding # 为一个指定的 ClusterRole 创建一个 ClusterRoleBinding configmap # 从本地 file, directory 或者 literal value 创建一个 configmap cronjob # Create a cronjob with the specified name. deployment # 创建一个指定名称的 deployment. job # Create a job with the specified name. namespace # 创建一个指定名称的 namespace poddisruptionbudget # 创建一个指定名称的 pod disruption budget. priorityclass # Create a priorityclass with the specified name. quota # 创建一个指定名称的 quota. role # Create a role with single rule. rolebinding # 为一个指定的 Role 或者 ClusterRole创建一个 RoleBinding secret # 使用指定的 subcommand 创建一个 secret service # 使用指定的 subcommand 创建一个 service. serviceaccount # 创建一个指定名称的 service account
Options: --allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats. --dry-run=false: If true, only print the object that would be sent, without sending it. --edit=false: Edit the API resource before creating -f, --filename=[]: Filename, directory, or URL to files to use to create the resource -k, --kustomize='': Process the kustomization directory. This flag can't be used together with -f or -R. -o, --output='': Output format. One of: json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file. --raw='': Raw URI to POST to the server. Uses the transport specified by the kubeconfig file. --record=false: Record current kubectl command in the resource annotation. If set to false, do not record the command. If set to true, record the command. If not set, default to updating the existing annotation value only if one already exists. -R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory. --save-config=false: If true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future. -l, --selector='': Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2) --template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview]. --validate=true: If true, use a schema to validate the input before sending it --windows-line-endings=false: Only relevant if --edit=true. Defaults to the line ending native to your platform. Usage: kubectl create -f FILENAME [options]
代码：
