Go package net/http for web scraping or spidering. Understanding of robots.txt

Home
Mini-Programs
- Cryptography in Go
- Data Structure and Algorithm
  - Heterogeneous
My Go Tutorial
- Cloud Computing (SaaS)SaaS (Software as a Service): This is the highest layer offering you can obtain as a cloud solution. A SaaS solution is when a fully functional piece of software is delivered over the web. You access SaaS solutions from a web browser
- Git
- Graphic Library
  - Game Engine
- Middleware
  - gorilla/muxPackage gorilla/mux implements a request router and dispatcher for matching incoming requests to their respective handler. The name mux stands for “HTTP request multiplexer”. Like the standard http.ServeMux, mux.Router matches incoming requests against a list of registered routes and calls a handler for the route that matches the URL or other conditions. The main features are: It implements the http.Handler interface so it is compatible with the standard http.ServeMux. Requests can be matched based on URL host, path, path prefix, schemes, header and query values, HTTP methods or using custom matchers. URL hosts, paths and query values can have variables with an optional regular expression. Registered URLs can be built, or “reversed”, which helps maintaining references to resources. Routes can be used as subrouters: nested routes are only tested if the parent route matches. This is useful to define groups of routes that share common conditions like a host, a path prefix or other repeated attributes. As a bonus, this optimizes request matching.
- Language
  - Functions_Closures
  - Struct TypeA struct is a sequence of named elements, called fields, each of which has a name and a type. Field names may be specified explicitly (IdentifierList) or implicitly (EmbeddedField). Within a struct, non-blank field names must be unique.
- MongoDB
- RedisRedis is an open source key-value store that functions as a data structure server.
- Regular ExpressionA regular expression is a pattern of text that consists of ordinary characters (for example, letters a through z or numbers 0 through 9) and special characters are known as metacharacters. This pattern describes the strings that would match when applied to a text.
- RPC- gRPC
Standard Library
- bufioPackage bufio implements buffered I/O. It wraps an io.Reader or io.Writer object, creating another object (Reader or Writer) that also implements the interface but provides buffering and some help for textual I/O.
- csvPackage csv reads and writes comma-separated values (CSV) files. There are many kinds of CSV files; this package supports the format described in RFC 4180. A csv file contains zero or more records of one or more fields per record. Each record is separated by the newline character. The final record may optionally be followed by a newline character.
- crypto/sha256
- html/templatePackage template (html/template) implements data-driven templates for generating HTML output safe against code injection. It provides the same interface as package text/template and should be used instead of text/template whenever the output is HTML. The documentation here focuses on the security features of the package. For information about how to program the templates themselves, see the documentation for text/template.
- io/ioutilPackage ioutil implements some I/O utility functions.
- log
- net/http
- regexp
- osPackage os provides a platform-independent interface to operating system functionality. The os interface is intended to be uniform across all operating systems. Features not generally available appear in the system-specific package syscall.
- strings
- text/templatePackage template implements data-driven templates for generating textual output. To generate HTML output, see package html/template, which has the same interface as this package but automatically secures HTML output against certain attacks. Templates are executed by applying them to a data structure. Annotations in the template refer to elements of the data structure (typically a field of a struct or a key in a map) to control execution and derive values to be displayed. Execution of the template walks the structure and sets the cursor, represented by a period ‘.’ and called “dot”, to the value at the current location in the structure as execution proceeds. The input text for a template is UTF-8-encoded text in any format. “Actions”–data evaluations or control structures–are delimited by “{{” and “}}”; all text outside actions is copied to the output unchanged. Except for raw strings, actions may not span newlines, although comments can. Once parsed, a template may be executed safely in parallel, although if parallel executions share a Writer the output may be interleaved.
Special Interest
Utilities and Tools
Debugging
Performance
Testing

https://support.google.com/webmasters/answer/6062608?hl=en

Categories: Uncategorized

M	T	W	T	F	S	S
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31

Go package net/http for web scraping or spidering. Understanding of robots.txt

Published by Lim Shan Kuo on December 11, 2019December 11, 2019

0 Comments

Leave a Reply Cancel reply

Uncategorized

Every Thing Is Important (Prioritise)

Uncategorized

Hello World in Function, Anonymous Function, First-Class Function.

My Go Tutorial

Using Map with Examples

Go package net/http for web scraping or spidering. Understanding of robots.txt

Published by Lim Shan Kuo on December 11, 2019December 11, 2019

0 Comments

Leave a Reply Cancel reply

Related Posts

Uncategorized

Every Thing Is Important (Prioritise)

Uncategorized

Hello World in Function, Anonymous Function, First-Class Function.

My Go Tutorial

Using Map with Examples