笔记本做网站服务器,如何建立小企业网站,饰品公司网站建设方案,玉雕网站建设目录
一. 前言
二. Kafka 硬件和操作系统#xff08;Hardware and OS#xff09;
2.1. 操作系统#xff08;OS#xff09;
2.2. 磁盘和文件系统#xff08;Disks and Filesystem#xff09; 一. 前言 Kafka 是 I/O 密集型而非计算密集型的框架#xff0c;所以对 CP…目录
一. 前言
二. Kafka 硬件和操作系统Hardware and OS
2.1. 操作系统OS
2.2. 磁盘和文件系统Disks and Filesystem 一. 前言 Kafka 是 I/O 密集型而非计算密集型的框架所以对 CPU 的需求是各个指标里最宽松的消耗CPU 的点主要在于消息的压缩和解压缩。一个 Kafka Broker 节点往往要承载许多个 Topic Partition 并与许多个 Producer/Consumer 交互所以并行度核心/线程数要比单核性能频率更重要。 一般来讲单节点 8C/16T主频 2GHz 以上按 Broadwell 架构计就可以满足小型生产环境负载比较重的集群可以配到 12C/24T 甚至 16C/32T。注意根据 CPU 规格的不同Broker 的num.network.threads 和 num.io.threads 参数也要适当改变。
二. Kafka 硬件和操作系统Hardware and OS 原文引用We are using dual quad-core Intel Xeon machines with 24GB of memory. You need sufficient memory to buffer active readers and writers. You can do a back-of-the-envelope estimate of memory needs by assuming you want to be able to buffer for 30 seconds and compute your memory need as write_throughput*30. 我们使用的是具有 24GB 内存的双四核 Intel Xeon 机器。 您需要足够的内存来缓冲活动的读写器。假设您希望能够缓冲30秒并将您的内存需求计算为write_throughput*30您可以对内存需求进行粗略估计。 原文引用The disk throughput is important. We have 8x7200 rpm SATA drives. In general disk throughput is the performance bottleneck, and more disks is better. Depending on how you configure flush behavior you may or may not benefit from more expensive disks (if you force flush often then higher RPM SAS drives may be better). 磁盘吞吐量很重要。我们有 8x7200 rpm SATA 驱动器。一般来说磁盘吞吐量是性能瓶颈磁盘越多越好。根据您配置刷新行为的方式您可能会从更昂贵的磁盘中受益也可能不会从中受益如果您经常强制刷新则 RPM 更高的 SAS 驱动器可能会更好。
2.1. 操作系统OS 原文引用Kafka should run well on any unix system and has been tested on Linux and Solaris. We have seen a few issues running on Windows and Windows is not currently a well supported platform though we would be happy to change that. Kafka 应该在任何 unix 系统上都能很好地运行并且已经在 Linux 和 Solaris 上进行了测试。 我们在 Windows 上看到了一些问题Windows 目前不是一个受支持的平台尽管我们很乐意改变这一点。 原文引用It is unlikely to require much OS-level tuning, but there are three potentially important OS-level configurations: File descriptor limits: Kafka uses file descriptors for log segments and open connections. If a broker hosts many partitions, consider that the broker needs at least (number_of_partitions)*(partition_size/segment_size) to track all log segments in addition to the number of connections the broker makes. We recommend at least 100000 allowed file descriptors for the broker processes as a starting point. Note: The mmap() function adds an extra reference to the file associated with the file descriptor fildes which is not removed by a subsequent close() on that file descriptor. This reference is removed when there are no more mappings to the file.Max socket buffer size: can be increased to enable high-performance data transfer between data centers as described here.Maximum number of memory map areas a process may have (aka vm.max_map_count). See the Linux kernel documentation. You should keep an eye at this OS-level property when considering the maximum number of partitions a broker may have. By default, on a number of Linux systems, the value of vm.max_map_count is somewhere around 65535. Each log segment, allocated per partition, requires a pair of index/timeindex files, and each of these files consumes 1 map area. In other words, each log segment uses 2 map areas. Thus, each partition requires minimum 2 map areas, as long as it hosts a single log segment. That is to say, creating 50000 partitions on a broker will result allocation of 100000 map areas and likely cause broker crash with OutOfMemoryError (Map failed) on a system with default vm.max_map_count. Keep in mind that the number of log segments per partition varies depending on the segment size, load intensity, retention policy and, generally, tends to be more than one. 它不太可能需要太多的操作系统级别调整但有三种潜在的重要操作系统级别配置
文件描述符限制Kafka 对日志段和开放连接使用文件描述符。如果一个 Broker 托管许多分区那么除了 Broker 建立的连接数之外还需要考虑该 Broker 至少需要(number_of_disparations)*(partition_size/segment_size) 来跟踪所有日志段。我们建议至少100000个允许的文件描述符作为 Broker 进程的起点。注意mmap() 函数为与文件描述符过滤器相关联的文件添加了一个额外的引用该文件描述符过滤器不会被该文件描述符的后续close() 删除。当不再有到该文件的映射时将删除此引用。最大套接字缓冲区大小可以增加以实现数据中心之间的高性能数据传输如本文所述。进程可能具有的内存映射区域的最大数量也称为 vm.max_map_count。请参阅 Linux 内核文档。在考虑 Broker 可能具有的最大分区数时您应该密切关注此操作系统级别的属性。默认情况下在许多 Linux 系统上vm.max_map_count 的值约为65535。每个分区分配的日志段都需要一对索引/时间索引文件每个文件占用1个映射区域。换句话说每个日志段使用2个 map 区域。因此每个分区至少需要2个映射区域只要它承载一个日志段即可。也就是说在一个 Broker 上创建50000个分区将导致100000个映射区域的分配并可能导致具有默认 vm.max_map_count 的系统上出现 OutOfMemoryError映射失败的Broker 崩溃。请记住每个分区的日志段数因段大小、负载强度、保留策略而异通常情况下往往不止一个。
2.2. 磁盘和文件系统Disks and Filesystem 原文引用We recommend using multiple drives to get good throughput and not sharing the same drives used for Kafka data with application logs or other OS filesystem activity to ensure good latency. You can either RAID these drives together into a single volume or format and mount each drive as its own directory. Since Kafka has replication the redundancy provided by RAID can also be provided at the application level. This choice has several tradeoffs. 我们建议使用多个驱动器以获得良好的吞吐量而不是与应用程序日志或其他操作系统文件系统活动共享用于 Kafka 数据的相同驱动器以确保良好的延迟。您可以将这些驱动器 RAID 到一个卷中也可以格式化并将每个驱动器装载为自己的目录。由于 Kafka 具有复制功能RAID 提供的冗余也可以在应用程序级别提供。这个选择有几个折衷方案。 原文引用If you configure multiple data directories partitions will be assigned round-robin to data directories. Each partition will be entirely in one of the data directories. If data is not well balanced among partitions this can lead to load imbalance between disks. 如果配置多个数据目录分区将被分配给数据目录。每个分区都将完全位于其中一个数据目录中。如果分区之间的数据没有很好地平衡这可能会导致磁盘之间的负载不平衡。 原文引用RAID can potentially do better at balancing load between disks (although it doesnt always seem to) because it balances load at a lower level. The primary downside of RAID is that it is usually a big performance hit for write throughput and reduces the available disk space. RAID 可能在平衡磁盘之间的负载方面做得更好尽管它似乎并不总是这样因为它在较低级别上平衡负载。RAID 的主要缺点是它通常会对写入吞吐量造成很大的性能影响并减少可用磁盘空间。 原文引用Another potential benefit of RAID is the ability to tolerate disk failures. However our experience has been that rebuilding the RAID array is so I/O intensive that it effectively disables the server, so this does not provide much real availability improvement. RAID 的另一个潜在好处是能够容忍磁盘故障。然而我们的经验是重建 RAID 阵列是 I/O 密集型的它会有效地禁用服务器因此这并不能提供太多实际的可用性改进。
